Resolving the tension between beginner friendliness and advanced usage

[MateuszKowalewski]

@Jasper-M

But that would mean that if your “core” uses advanced features, then your user facing API will also have to be labeled “advanced” because it depends on the core.

I’m indeed not sure about that.

As @jducoeur mentioned already in one of the first replies, it depends on how you define “usage”.

I think just importing something has no effect as such on this feature. But you don’t import stuff just for fun of course. You use this stuff than somehow. I think whether the code part where you imported something becomes “advanced” (or “standard” in case of “basic mode”) depends on what language features you actually use in your code than.

That’s why I’ve said that almost every (serious) codebase will have “advanced” dependencies. If this would make all your depended code automatically also “advanced” the whole idea would be moot in the first place.

That’s a bit too “viral”…

But yes, this needs some fleshing out. I’ve heard here are some smart people around… So details can be worked out. What’s the first step is to sell the basic idea.

I just didn’t expect that we move so fast into “implementation details”. I thought it would take some more time to convince people about the basic principle, as quite some people see some mechanism which is somehow similar to “language imports” very skeptical; imho rightly so, as “language imports” just don’t work! But my proposal is exactly for this reason different in some key parts.

@Jasper-M

Most of the perceived complexity these days seems to come from “effect systems”, but ironically those systems don’t necessarily depend on complex or advanced language features. Their cores are often implemented in very low level code for performance reasons, they don’t require macros and as long as you stick to a concrete implementation (code against cats.effect.IO instead of cats.effect.Async) you don’t even need higher kinded types or implicits.

That’s actually true.

I also don’t think that “naked IO” is complex as such. (If you would hard-code stuff to IO and leave out the than unnecessary type-parameter everywhere in “IO code” this would result actually in quite simple and straight forward APIs. But doing such a hard-coding is against the ideology there, so this won’t happen I guess, even it would make the whole Cats framework quite approachable even for average people. But imagine that it would become a selling point of a lib to be “standard” or “basic” Scala. Maybe someone would build than for that reason a kind of “overlay” lib on top of Cats, just hardcoded to IO, and start “selling” this lib with the “basic” Scala tag. Such a well visible tag in the ecosystem could become a kind of “sales argument”! And it could motivate initiative for some libs for such a move like described).

Also I’ve mentioned already that “defining ‘advanced’” is quite a task on its own!

We have mostly only anecdotes about what “advanced” Scala is… But we should imho go to the real source: Learners and instructors. Gather some proper data form there, and not define “advanced” ad hoc out of the gut feeling of some.

[MateuszKowalewski]

@phrmoy

If not, that hot potato will not land on my hand when some disgruntled Java developer who is not too keen on Scala tells an executive that we did not hit the milestone because I had all of my offshore engineers spend months on picking up Scala.

So you say that Scala is hard to pick up, even for already experienced engineers? Do I get this correctly?

Why do you think so?

What could make the language less difficult to get picked up?

Why do you think experienced Java devs would need months (!) to get productive with Scala? (Even you can write “Java with Scala syntax” just the next day after picking it up).

@phrmoy

will offer that edge to other when it’s justifiable to do so. Often, it’s not.

Why? That’s actually core of the question here.

@phrmoy

Funny enough, out of all of my degrees, the less technical one is the only that really taught me about economy of scales, and Scala just isn’t part of that, all other options considered.

Scala doesn’t “scale”? Do I understand this correctly?

What do you mean by that?

Sorry for all the questions, but it’s indeed quite interesting to have some kind of “outside” view!

The people actively participating here have usually many years of Scala experience. This may blur the vision in some regards…

[morgen-peschke]

@MateuszKowalewski

First of all this isn’t about “locking anything away”. Maybe I still didn’t explain this sufficiently, but the change on the language side would be a new kind of marker (in the spirit of “language imports”, but very coarse-grained, so no “information noise” or micromanagement).

It definitely would be used to lock away features though, particularly if these capabilities are provided:

@MateuszKowalewski

More interestingly, “advanced” Scala could be restricted to only some modules of an application. Organizations could also set limits on the relative number of such “advanced” modules in a code base.

@MateuszKowalewski

Such a tool would also enable to enforce architectural decisions.

This would definitely increase the number of Scala specific questions I’d need to ask during an interview, which would be a bit annoying.

[MateuszKowalewski]

I fully understand this sentiment.

The proposal as such doesn’t “lock away” anything in its own, but downstream users could use it for that.

That’s not only true, it’s in some part intended.

In the end it’s about striving a balance between the needs of two groups which have maximally opposing views on this topic.

There is the “less power is better” group. My conclusion is that they have valid arguments—even I’m personally in the “opposing” group.

I’ve designed the idea in a way that “locking away features” would also be a hard sell, exactly because it shouldn’t become “the norm”! (Even I would envision that having those tags proper could be a selling point; depending on your target audience). As this categorization of features is maximally coarse-grained you have only almost just an “all or nothing” option. You can of course lock away whole “advanced” (or even “standard”) Scala. Sure. But this would cripple your possibilities substantially. Doing that wouldn’t be something you would apply therefor to whole codebases. You would do that only for parts where it actually makes sense. And even there it wouldn’t be a lighthearted decision.

But especially large organizations have people around with all kinds of expertise level. Being able to steer some “big picture” guidelines is a necessity for them.

Also there is the tension around making Scala on-boarding simpler for newcomers. Having a kind of “sandbox mode” (where tooling could tell you where the sandbox is exactly) could help imho in this regard.

Also, like mentioned, the main purpose isn’t making it possible to “lock away” large parts of Scala. The intent is first and foremost to make the “language level” part of machine readable documentation / meta-data. So it could become a data-point when looking on projects, like for example GitHub stars are. If someone looks for a “simple lib” they could filter for example Scaladex for such a meta-data tag. That you sometimes need something “simple” is a valid requirement. But this goes of course also the other way around. If you’re explicitly looking for something “advanced” just filter for the matching meta-data tag and you don’t need to look through all the libs / projects that turn out to be made just for simple use-cases. (Even the whole thing is more about language complexity. But this goes hand in hand in parts. And the concrete definition of the categories is still to be given, but that’s imho one of the last steps).

I’ve provided also already an example why having such meta-data, and metrics based on this data, is good for everyone. As an advanced user you could explicitly check whether you’re allowed to produce or use more advanced features. This could make some discussions easier. And again, being coarse-grained helps here: If someone allows “advanced” Scala, they can’t come afterwards with “but this and that feature we don’t want to see here”. It’s an all-or-nothing thing! Stupid discussions why you may use this but not that would become easy to resolve: “You’ve said, ‘advanced’, I’m using ‘advanced’. If you like to limit it, it needs to be geared down to ‘standard’. But than we can’t also use this and that and that. Do you really want it this way?”

So I wouldn’t expect that “hard lock-downs” would become the norm. The trick is that it works both ways…

[MateuszKowalewski]

One more thing: I’m not even sure that using FP concepts counts as “advanced” as such.

This is one of the parts that needs to get figured out. Based on data.

So please take all examples of what constitutes one of the named categories with a grain of salt. This is imho something that’s still completely undefined here. We had some dummies around until now. Not more.

[morgen-peschke]

@MateuszKowalewski

If someone allows “advanced” Scala, they can’t come afterwards with “but this and that feature we don’t want to see here”. It’s an all-or-nothing thing! Stupid discussions why you may use this but not that would become easy to resolve: “You’ve said, ‘advanced’, I’m using ‘advanced’. If you like to limit it, it needs to be geared down to ‘standard’.

Currently this is handled in code reviews, and missing out on this bypasses a really effective tool for teaching junior devs what a particular language feature is good for, when it can be used effectively, and when it will cause more trouble than it’s worth.

@MateuszKowalewski

If someone looks for a “simple lib” they could filter for example Scaladex for such a meta-data tag. That you sometimes need something “simple” is a valid requirement.

I’m not sure this would actually be a useful filter, as the simplicity of the language features used in a project don’t really say anything useful about the amount of domain knowledge someone would need to work on that project.

[MateuszKowalewski]

@morgen-peschke

Currently this is handled in code reviews, and missing out on this bypasses a really effective tool for teaching junior devs what a particular language feature is good for, when it can be used effectively, and when it will cause more trouble than it’s worth.

I’m very unsure how anything in my proposal could replace code reviews in any way.

You asked about “tooling enforced guidelines” previously, didn’t you?

@morgen-peschke

I’m not sure this would actually be a useful filter, as the simplicity of the language features used in a project don’t really say anything useful about the amount of domain knowledge someone would need to work on that project.

I’ve relativized this part with the last addition. (I didn’t like to rewrite the post in a meaning altering way, so sorry for the confusion!)

Yes, this proposal is about language complexity. So the example didn’t match exactly.

This kind of filter on a library / project index would be mostly helpful to find something that matches the current language skill level. It could give only very indirectly some info about whether a lib or project is “simple” as such; you’re right. But you can have libs with the same required domain knowledge but very different requirements for the proficiency on the language level! Having a filter for that would still make insofar a lot of sense.

[phrmoy]

@MateuszKowalewski

So you say that Scala is hard to pick up, even for already experienced engineers? Do I get this correctly?

Why do you think so?

What could make the language less difficult to get picked up?

Why do you think experienced Java devs would need months (!) to get productive with Scala? (Even you can write “Java with Scala syntax” just the next day after picking it up).

So when I say Scala, I mean the entirety of the Scala development environment, which consists of the language, the build tools (plural), and editor support (when I just arrived IntelliJ was not even usable so I had to get started on VSCode, which was a huge blow being hooked to JetBrain for so long), then finally, libraries (here com-lihaoyi · GitHub really helped, luckily enough for me, they are just about stable or are soon to become).

So yes, and honestly I think there is some expectation that a JVM engineer will pick up all of the above, without issues. And I don’t really see why that expectation exists (aside from how down the road having an intricate understanding of how the JVM works may be beneficial to you). I digress though, I don’t really mean to focus on the particular transition from Java to Scala. But it’s good to note that just saying something like “writing Java with Scala syntax” does not begin to start doing justice to the hurdles you go through to make such a major switch (delivering work proficiently and efficiently) because the syntax is just syntax, never was the issue. Now, to be fair, if you’re coming from a dynamically typed language like Python, then yes, it can get much worse for you.

Now how to improve things? I spent a considerable amount of time setting up my environment, and now it’s passable. This includes understanding SBT (multi-projects), bloop, scalafmt, compiler flags, that were nowhere to be found but in the actual source code (help I got from very kind people willing to help on Discord), JVM flags, and VSCode stuff (clearly not a Scala thing, but I had to do it - in fact, I even had to hack into the Metals plugin to make things work in my machine). Imagine, I actually created a VSCode plugin (Yes, JavaScript), I kid you not. But why would you do that you may ask? Because I had to get things working (in all fairness I have encountered much help from people working on these tools and I appreciate that). But when you gotta go you gotta go, I had to just fix things and get going. Then I also needed to figure out my flat jar binaries and automate that with respect to VSCode, and Graal Native (sure, maybe an extra thing, but I am down the rabbit hole already right? I actually needed my start-up times descent for some scripts), and automate that with respect to VSCode. Good Lord, then it came time for me to handle some JSON and CSV, serialize/persist/cache some data with ease, and parse arguments. So I went on to build myself some light abstractions (it was a good learning experience too). Obviously, this has already become a rant, apologies but hopefully it reveals the onboarding experience a little. Maybe you’re thinking, you have to do for other languages too! Maybe you’re right, but here I found unpolished and fragmented ways of doing things that really made my experience setting up long and exhausting, and/or things that just were there and I had to just go out and kinda put together quickly. This is clearly a newcomer’s thing but has little to do with the Scala 3 language itself, I take no issue with Scala syntax and some choices made in the Scala 3 language, even if I may find it odd.

I can share more later if this is helpful feedback.

Pedro

[MateuszKowalewski]

@phrmoy

I can share more later if this is helpful feedback.

Yes, thank you! That’s helpful.

@phrmoy

Obviously, this has already become a rant, apologies but hopefully it reveals the onboarding experience a little.

If it’s a rant than a very polite one.

I admit, your on-bording experience sounds quite wild. I’m not sure this is the norm. I hope it is not!

What I read between the lines is that something like a “batteries-included toolkit” would have been really useful in your case.

I guess you’ve been a little bit too early. Scala has now something like that:

Introduction

Switching IDEs is of course a great PITA. I guess it’s just like it is.

But VSCode works imho at the moment very well for Scala. For me it makes less trouble than IntelliJ. Still IntelliJ is superior with things unrelated to the Scala support, I admit.

The other thing seems to be still subpar documentation. Quite a lot of the things you describe are actually meant to “just work”, and they do usually. But you need to know which kind of special sauce you need. This knowledge is indeed kind of “tribal”. There is no central place for all of that. The Scala site has infos about Scala, and those are quite good by now. But they won’t tell you about the broader ecosystem most of the time. (Otherwise someone would cry that Scala endorses this but not that lib / framework / whatever; and this is actually understandable).

So you need to puzzle info together.

I’m not saying that this is substantially different in other languages. But elsewhere you have most of the time large framework that will form a kind of ready to run environment and have comprehensive documentation, or you have at least some form of “end to end” guides on some topics. Scala could be stronger on both points, that’s true.

Maybe some feedback regarding some concrete issues, but I don’t know of course whether you found it all already by yourself.

• For compiler params a good starting point is the tpolecat sbt plugin.

• For fat jars the sbt assembly plugin is helpful.

• For Graal Native Images you can use the sbt native packager plugin.

• Actually everything build related: There is likely a SBT plugin for that…

Regarding stuff like JSON and CSV, serialize/persist/cache, it’s all there. Even multiple times, because in here nobody agreed on a “std. solution”. Not different to other languages, only that Scala again doesn’t have a “quasi standard” like a lot of languages have. It’s more diverse.

I guess again better (visible?) documentation would help…

May I ask with which stack you went? How was it in comparison to other languages?

I agree with the part that stuff is “fragmented” and that Scala heavily misses proper frameworks (I hope the “toolkit initiative” will help here!), but “unpolished”? A lot of libs work imho much better in Scala than elsewhere. Of course, after you puzzled all the needed components together by yourself, which is imho indeed annoying. (I still have to test out that toolkit thingy myself… )

Still it’s true that getting standard stuff quickly done is often simpler in other languages. But that’s imho a “framework problem”. For example the Cats and ZIO “ecosystems” have amazing components, but it’s still simpler to pull off for example a simple JSON web-API with something like Spring Boot. Just because the later automates so much for you. The components may be much weaker but they get assembled by the press of a button… That’s hard to beat without proper language and tooling support. (And Scala 3 lacks here…)

@phrmoy

I take no issue with Scala syntax and some choices made in the Scala 3 language, even if I may find it odd.

I’m not sure I understand fully. If you have qualms about Scala as a language this is the right forum. People are listening. Mr. Odersky himself was actually interested in your opinions. He and his team are trying hard to improve the on-boarding experience and make the language friendlier for newcomers. So feedback is very much appreciated, I think.

[phrmoy]

Thank you for being so kind in receiving all my rather harsh criticism, and for sharing the resources, I have used them all except for GitHub - typelevel/sbt-tpolecat: scalac options for the enlightened which I found a little lacking for Scala 3 so I built devised my own set of flags. Will be sure to check out the Toolkit tutorials to make sure I did not miss on anything good.

As far as the stack I went with, I have different needs. For research, I built some light abstractions over com.lihaoyi to accomplish simple file and data handling, requests, and serialization plus some automatic derivations from circe for JSON. com.typesafe config and logging. Will still settle on a lib to handle postgres when time comes (Skunk probably), and use mongodb’s Scala SDKs when time comes. Will use a comb of Python’s scripting and Scala-driven notebooks for viz. As needed, will use Scala parallel lib for crunching some data on my laptop. For parallel tasks, will use either vanilla Scala’s Futures and or Akka for heavier asynchronous work. Spark for more serious data parallel work as needed. When it comes time to build the client facing the broker, will decide between Akka or ZIO. I don’t have to jump on the functional train, but I will if it delivers better value (it might).

I don’t think the libs are unpolished, com.lihaoyi works great. I sort see Scala and its ecosystem right now as this amazing unpolished precious stone still. It feels as though it’s my job to carve it out and let it shine as needed. I don’t think that’s a bad thing necessarily, it just takes a lot of work and good dose of stubbornness not give up at the first hiccup, or several first hiccups.

No qualms with Scala as a language, if I am putting in the work is because I think Scala is a great language, somethings are just different such as allowing language features via imports, which I don’t mind. But I will bite, maybe I just we had fibers baked into the language? Ok probably out of my depth now. Thanks for listening to my feedback and asking questions.

Pedro

[sideeffffect]

The existence of GitHub - typelevel/sbt-tpolecat: scalac options for the enlightened is a great example of symptom of a problem.

Why would such tool even need to exist? A language/compiler should be helpful out of the box.

The fact that people need to go out of their way and create/adopt tools like this is what makes Scala more difficult to adopt. Jumping through hoops like this is not fun (and not even obvious to beginners).

[nafg]

You can get the upickle behavior with Circe result by just doing .toTry.get

[nafg]

FWIW the new build tool https://bleep.build has those flags on by default when you create a new project, and otherwise it’s just a matter of setting strict: true.

Switching most projects to Bleep would probably make a huge difference in the perceived complexity of Scala.

That said, forget about all the linting flags, even -deprecation and other basics are off by default. I think this is a legacy from javac and maybe other languages. But it’s a terrible default. (I don’t even know why there would even be a flag to turn it off, other than the general -Wconf.)

The question is, can we change it? Maybe in 3.4.0?

[markehammons]

Yeah there’s ways, but my point was that the complexity of Circe vs upickle is related to design choices. Circe by default makes you deal with the fact that parsing errors occurred. It’s a good strategy for building airtight programs. Upickle by default just gives you the result and lets an error be thrown if parsing failed. That design is a good choice for simple, quick programs.

These kinds of complexity due to design choices have nothing to do with advanced features and everything to do with what the library is for and what it’s trying to do.

[markehammons]

Generally I like the warnings turned on by this plugin, but I’ve been souring on -werror. As a metals user, -werror can make metals unusable for a long time when I’m prototyping or doing some other work where unused stuff, dead code, etc isn’t a major problem.

Maybe the linting options should be on by default in the future, but -werror should definitely be something that’s bought into.

[som-snytt]

-Werror is more of a CI backstop.

I don’t think the compiler should emit any warnings, including deprecation.

Third party tooling should handle deprecation policy, unused element policy (such as how long they are ignored in a dirty workspace while editing), etc.

The out-of-the-box for warning options is -Xlint. The tpolecat list was really partial unification, and maybe something else to tune the Typelevel fork. Probably -Wvalue-discard.

Maybe “actionable diagnostics” will let the compiler focus on correctness, while other tools take action on secondary issues. The trick is to get the diagnostic to the user before they waste time figuring out why their code doesn’t work yet.

Edit: What I came here to say was that Java sounds a lot like Scala on this issue of balancing new features and beginner user experience.

[nafg]

@som-snytt

-Werror is more of a CI backstop.

True. Although, not just CI but other types of calling scripts.

In fact, IMO the fact that it’s a compiler flag is a kludge. It would be more elegant if we could tell the build tool “return a nonzero exit code if there are warnings.” That would be more in line with the goal than the current practice of telling the compiler to pretend warnings are errors.

One difference would be that if you have one module that depends on another, you could get the problems with the downstream module if the upstream module only has warnings. In that sense, the difference between errors and warnings is not “is this really unacceptable,” but “is the code so incoherent that the compiler can’t figure out what code to emit so no point in continuing.”

[RichType]

My own view on the complexity issue, possibly prejudiced and misinformed as I’ve never been an insider to Scala’s core ecosystem development. But for what its worth:

It seems to me that in its earlier days Scala had a significant influx of people who wanted Haskell on the JVM. Presumably Clojure didn’t cut the mustard for them and they chose Scala as the least worst option. These people were often highly intelligent, skilled and made very significant contributions to the development of the Scala eco system. The problem was that they saw anything that didn’t contribute to creating Haskell on the JVM as not merely complexity, but unnecessary complexity. This included the whole class inheritance system.

This was a big problem because inheritance is fundamental to Scala. Scala sought not merely to take the inheritance systems from Java and other languages, but sought to significantly expand the capabilities and use cases of inheritance. This I think led to increasing frustration even anger by these Haskell programmers. I think many thought that given time @odersky and the Scala Community would see the light and be recruited to their project of making Scala into Haskell on the JVM. I think some became increasingly unhappy when it dawned on them that this wasn’t going to happen.

This looked very bad from a marketing perspective, because champions for other competing languages could say, “Look even Scala Programmers say Scala is too complex.” And yes much as people say otherwise, programming languages are in competition with each other. For small experimental languages confined to academia, the more the merrier approach may work. But languages that wish to break out of academia are in competition with each. More resources and developers for Scala means less resources and developers for some other programming languages.

So for these reasons I never felt that it was wise to accept the criticisms of Scala’s complexity as made in good faith, whether they were made from inside or outside the Scala Developer community.

[morgen-peschke]

@RichType

So for these reasons I never felt that it was wise to accept the criticisms of Scala’s complexity as made in good faith, whether they were made from inside or outside the Scala Developer community.

I think that there’s an important distinction to be made between, “this criticism does not align with my view of what Scala should be” and “this criticism was made in bad faith.” These are very different things.

[phrmoy]

Alright.

So as an outsider, I see these re-ramblings of the past as unproductive and uninteresting.

Do you really want to further language features while allowing beginner’s friendliness?

It’s actually quite simple. Let beginners focus on applications & industry and veterans on language features.

Please take a look at https://fsharp.org/ (this is pure gold). Look under sections “Use” and “Guides”. Look at how things are broken down by industry and how I can start planning a project without understanding F#'s most intricate features (I am sure there are some, but guess what I care more about? I will answer: software & community discoverability, maturity, robustness).

I have gone through our website, and we are completely missing F#'s industry focused “Use” and “Guide”. Completely. (our Library Index is not it).

When I visited https://fsharp.org/ do you think I went to “Learn” first? Nope. I went to Use and Guides (for the industries I care about). And right in those, I checked the repos for the projects and looked for number of likes, number of contributors, latest commit, and noted if they reached 1.0. All of the above allows me assess if I have a viable project path with F# in minutes.