I won't call myself an expert in visual novels, but I have played my fair share and I'm always fascinated by the stories that can be told with it. Visual novel is an interesting medium, sitting nicely between more interactive video games and, like, novels with illustrations. The very best of them, just like the very best of anything, make the best use of its medium to tell the story it wants to tell.
This is just a personal recommendation list in no particular order. If a friend of mine asks me about how to get into visual novels, these are the ones I'll recommend.
This list only contains visual novels originally written in English. Translations can be excellent, but as a wise man once said, reading a translation is like taking a shower with a raincoat on. It's not necessarily a bad experience, but it's definitely different, and for this list I'd rather you have the full experience of a visual novel without a raincoat.
Note also that this doesn't contain text-only Interactive Fiction. That's a different list that I may or may not get around to writing too.
Freeware
SC2VN
Link: https://teameleveneleven.itch.io/sc2vn
A linear visual novel about someone moving to Korea to become a professional e-sport player. You don't have to be familiar with e-sport or StarCraft II to understand it, though it's definitely meant to make you understand it. It's got solid characters and is a good look into a strange but very definitely real world.
The Shadows That Run Alongside Our Car
Link: https://loxrain.itch.io/shadows

A short, linear visual novel taking place in a single area with only two characters. It's done very well, with just the right polish in every line for maximum emotional impact.
The Elevator
Link: https://cyanidetea.itch.io/the-elevator
Similar as the previous one, and half of this one's developer is the same as the above. This one is also a short, linear, visual novel with only two characters. Have a bit more mystery/thriller/horror feel than the previous one.
first date/can't relate
Link: https://ub4q.itch.io/first-date

A DIY dating sim: choose who you are and choose who you want to partner with. Very well written and, apparently, very Australian? Has a genuinely warm underlying story. Takes only an hour or two experience it all.
Save the Date
Link: http://paperdino.com/save-the-date/
A short visual novel, but absolutely not linear. You're trying to take someone out on a date, but things keep going wrong. Fortunately you have the power of save-scumming at your side. It's an incredibly clever homage to the genre, gleeful in its destruction of the fourth wall.
Cinderella Phenomenon
Link: https://dicesuki.itch.io/cinderella-phenomenon

A fairly long otome game (otome game: a game where you're a girl and you get to choose between a couple Very Handsome bachelors) based on fairy tales. It's surprisingly thoughtful and intricate, and the romance isn't as heavy-handed as other otome can get. It's wild that a game of this size is free.
Every game made by Zephyo
Link: https://zephyo.itch.io/
Angle He aka Zephyo makes absolutely beautiful short visual novels about, among others, relationships and mental health. I've only tried He Beat Her, You Left Me., and I woke up next to you again, and they're definitely worth your time.
Freak-Quency
Link: https://www.deviantart.com/sakon04/art/Game-Freak-Quency-full-version-available-502435546
Full disclaimer: I'm a friend of the friends of the devs, and it's locally-made, so I'm obligated to recommend this to you. It lacks a bit/a lot of polish in both art and writing, but it's free and for that price, the size is immense. It plays like a Persona game with its day-based stat-raising social sim gameplay, just without the battles. It's also a lot like a Persona game in it being urban fantasy and featuring high school characters and life. The story's interesting enough, the characters are fun, it's a 7/10 at least.
don't take it personally, babe, it just ain't your story
Link: http://scoutshonour.com/donttakeitpersonallybabeitjustaintyourstory
This is pretty ancient, actually. I think this was the first story I've read with openly and casually gay drama. Very well-written. Pretty linear. More slice-of-life and an examination of technology and teenage drama than a story with an actual plot, but I enjoyed it.
Juniper's Knot
Link: https://dischan.co/downloads/#junipersknot
Link (Steam): https://store.steampowered.com/app/687590/Junipers_Knot/

A short, linear, visual novel that was probably my gateway into indie Western visual novels, and maybe indie games in general. It's short and sweet. And I know this list isn't for translations, but if you can read Indonesian, the Indonesian translation is absolutely top-notch, I was floored by it.
RE: Alistair++
Link: http://www.sakevisual.com/realistair/
A short otome game. Like, if you're trying to explain to someone what an otome game is, this game is definitely the one you've got to show. It's nice and fun and short.
Magical Otoge series made by batensan
Link: https://batensan.itch.io/

batensan made very lovely, very fluffy, otome visual novels. These games are just pleasant pillows to lie back on, sweet enough to give you diabetes, but also feels genuine and heartfelt, albeit with some stumbles at times. The first two games are free but are already decently-sized. I recommend giving at least Magical Otoge Ciel a try.
Doki Doki Literature Club
Link: https://ddlc.moe/
I'll just be direct: this is a horror game. It should have way more content warning on its front page than it already does. It's overall Excellent as a horror. It's Very Good as an examination of mental health for the first third, which was more a slice-of-life fare, and then just throws everything into the bin afterwards. It's Terrifying. I wrote a bit about it in this blog a long time ago. Probably shouldn't be your first visual novel.
Y;N (Yin)
Link: http://mirrowdothack.com/yin.html
Link mirror (direct download): https://drive.google.com/open?id=1AC6x2RXRLFdcYiCSiqwVpbhs0HcbddTB

This is a personal favourite. It's dark and surreal and weird, but still grounded in a way. It's soft horror, but it's mostly a mystery, not a thriller. I love the characters a whole lot. It ends in a bit of a sequel hook but it's unlikely to get continuation of any kind. Definitely flawed, but I've never played anything quite like it before.
Paid
DIVINATION
Link (Itch): https://mojiken.itch.io/divination
Link (Steam): https://store.steampowered.com/app/1180250/DIVINATION/
A short visual novel with an interesting interface and choice system. You're a fortune teller in a distant future where humans and robots live together, and your prediction are always 100% true. It explores the "are sapient robots human" conundrum in an interesting way, though I think it doesn't take the concept far enough. It might leave a lot to be desired, especially since it's a paid game, but for only $2 it's definitely worth a try.
Analogue: A Hate Story + Hate Plus
Link: https://store.steampowered.com/sub/32136/
This is an epistolary story: you're investigating an abandoned space ship by reading through its data logs, which were written like reports or diary entries. Your main goal is figuring out out what happened to the people who used to live there. There's a cute AI accompanying you, but the story itself is quite intricate and thoughtful, and a lot more character-based than the premise might imply.
Hate Plus is the sequel, featuring the same "reading datalogs" gameplay. I like it better than Analogue, but you definitely have to play Analogue first.
An Octave Higher
Link: https://store.steampowered.com/app/352780/An_Octave_Higher/

The worldbuilding here is fantastic, taking place in a steampunk 19th century powered by both magic and tech. But instead of exoticising the setting, it's used to talk a lot of about labour issues and colonialism. The characterization and the plot itself are, in my opinion, kind of weak, but the writing is good and the the worldbuilding is really what it's all about here.
VA-11 Hall-A
Link: http://waifubartending.com/
I'm sure this one shows up in every visual novel list. It's cyberpunk, but instead of siding with some street samurai or hacker fighting an evil mega-corp, you're just a bartender serving drinks to said street tech samurais and hackers while listening to their stories. It's slice-of-life in an unusual setting. Very comfy.
Tangle Tower
Link: https://store.steampowered.com/app/359510/Tangle_Tower/

More of a point-and-click adventure game than a visual novel, but the dialogue interface is very much a visual novel and the game is about 80% dialogue. As a visual novel, it's a technical marvel. The way the sprite is animated and how it's perfectly tuned to the voice acting is *chef's kiss*. I wrote a fuller review of it here.
Amplitude
Link: https://store.steampowered.com/app/704680/AMPLITUDE_A_Visual_Novel/
Full disclosure: made by the same folks who made Freak-Quency. Also I got paid writing a tiny part of the game, so I have a personal stake in people playing this game. It's pretty much just like Freak-Quency, but more polished and with a different, somewhat longer story. Lemme just copy and paste the same blurb: "It plays like a Persona game with its day-based stat-raising social sim gameplay, just without the battles. It's also a lot like a Persona game in it being urban fantasy and featuring high school characters and life. The story's interesting enough, the characters are fun, it's a 7/10 at least."
Pyre
Link: http://store.steampowered.com/app/462770/Pyre/
[caption id="attachment_175148739" align="alignnone" width="800"]
Sir Gilman is a good boy and I love him[/caption]
This one should be familiar for those who like indie games; it's made by the same folks who made Bastion and Transistor. Pyre is described as a "party-based RPG" but it's really 30% a sports game (with RPG elements) and 70% a visual novel. It has fantastic worldbuilding, wonderful characters, and a connection between its story and gameplay to the point that every playthrough will always be personal to you. To describe the game in a silly way: it's power fantasy for bookworms, for introverts, for people who are shy and spend a lot of their time reading instead of socialising.
Arcade Spirits
Link (Itch): https://fictionfactorygames.itch.io/arcade-spirits
Link (Steam): https://store.steampowered.com/app/910630/Arcade_Spirits/
This game was part of my GOTY list in 2020 (the weird pandemic year). Let me just quote what I wrote "It might not hit all the beats for me, but it came at a particular time this year when I really welcomed its upbeat and unashamed we-can-get-through-this tone. It’s a visual novel about making friends and chasing your dreams and while narratively it treads familiar grounds, I really admire its energy. I remember playing this when I was feeling down, and it always made me feel better afterwards."
Red Embrace: Hollywood
Link (Itch): https://argent-games.itch.io/red-embrace-hollywood
Link (Steam): https://store.steampowered.com/app/904520/Red_Embrace_Hollywood/
This is some top-notch vampire fiction. If you've ever played any Vampire the Masquerade, or if you're even remotely interested in the aesthetic, this is definitely worth a try. Red Embrace: Hollywood is kind of like an otome in that you've got a choice of three boys, each with their respective stories over an overarching one, but it's also kind of not because you don't always get into a relationship with the boy you want. Also going against most other otome, the story can end in tragedies no matter what you do. As a romance game, though, it's incredibly well-written. I went from "lmao who is this weirdo" to "I love him and I will die for him and just seeing him around make me smile" within its 5-hour playthrough. I'll probably double that playtime as I try to get all the endings.
Amarantus
Link (Itch): https://ub4q.itch.io/amarantus
Link (Steam): https://store.steampowered.com/app/1941100/Amarantus/
Made by the same solo developer that made first date/can't relate above. I think its store pages do a good job of selling itself "Revolutions are messy. Relationships are messier." Out of every visual novels I've listed here, this one have the most naturalistic writing, which also extends to how it's presented and its pacing. Like, I'm sure internally there's some kind of stat system for the decisions you made, how close you are to each party member etc, but this is never shown so you're mainly feeling it out from what the characters say. It also has a lot of subtle body languages---eye movements, shifted pose, all that jazz--which is pretty impressive. I overall love this game and its ragtag band of friend, but it is also incredibly obtuse so make sure you're ready for not understanding a damn thing.
This is kind of a lot. Which one should I play?
Whichever in the freeware section is most interesting to you, but if you want my recommendation...
I'd say The Shadows That Run Alongside Our Car is a perfect starter. It's short enough to play in one sitting and it has a bit of branching, so it's not entirely linear. Juniper's Knot is a great short story, but it has no interactivity at all, so that's a good taster for completely linear VNs. first date/can't relate is excellent if you want to try something romantic. If you want to know more about otome games, then try RE: Alistair++, and if you dig that, you'll probably like Cinderella Phenomenon.
If you've always been skeptical of the "game" part of visual novels, then try Save the Date, DIVINATION, or Pyre. Each of these games does something interesting with their mechanics in such a way that it's impossible to be adapted straight into, say, a novel or a movie.
If you want something weird, try Y;N.
I think that's all. There's a couple of popular visual novels I omit because I haven't played them yet (Dream Daddy, Aviary Attorney, Hatoful Boyfriend, We Know the Devil, Heaven Will be Mine, to name a handful). And including non-originally-English visual novels will probably double the size of the list (Ace Attorney, True Remembrance, 428: Shibuya Scramble, Mystic Messenger, etc etc).
I'm always up for more good visual novels, so if you know any that I haven't put on the list yet (or if you think it's wild I haven't played We Know the Devil yet and that I should totally get to that), drop me a message.