Resolving the tension between beginner friendliness and advanced usage

Language Design
1

Since more or less forever there has been a kind of tension accompanying the design of the language between, on the one hand, what makes Scala a great choice for teaching and, on the other hand, the needs of advanced power users.

Both sides have very strong arguments, but the goals of both sides seem contradictory.

Compromises, as always, often leave both sides quite dissatisfied. That’s the nature of compromise.

Personally, I think neither side can be dismissed:

Success in the area of teaching is the be-all and end-all of a language’s spread and prevalence. If you want your language to grow, you have to make it easy for people to learn it. If you scare off newcomers, all is lost…

At the same time, Scala is especially successful among experienced and advanced developers. A small team of talented people who know what they’re doing can do wonders with all the powerful features Scala has to offer. Very often people choose Scala exactly because it offers features that are not available in more “industry-oriented” languages.

Scala is sort of the C++ of the JVM; for the good, and frankly for some of the bad. Powerful features come with complexity, and complexity clearly has a price. Be it just the problem of hiring sufficiently experienced people.

I would like to propose a partial solution, and a way to ease the tension between the two valid points of view.

In fact, it’s not even an entirely new idea at its core. A long, long time ago, somewhere I can’t remember, @odersky proposed something he called “language levels”. The idea was quickly rejected, for reasons I think are understandable. It was a bit over the top, and most importantly, not actionable in any way. I think the proposal included something like ten “language levels”, rather arbitrarily tied to specific language features. It would probably be impossible to write any real-world application without using up to the most “powerful” levels. Which would make the whole concept moot.

But the idea of “language levels” as such is actually quite good! It just needs to be simplified and made at least somewhat workable.

So I would like to propose the introduction of three semantic “language levels”: “basic”, “standard” and “advanced”.

  • “Basic” Scala would include everything needed for teaching beginners. It would leave out quirky and confusing parts. No magic. But enough to get things done. Maybe think “Python, non-expert level”.

  • “Standard” Scala would include besides everything in “basic” some of the more rough but commonly needed parts of Scala. Think “everything an average professional developer needs on a daily basis”.

  • “Advanced” Scala would include “standard”, but also the more obscure parts of the language not everyone is familiar with, and of course all of the most advanced features.

I don’t want to discuss in detail which feature belongs where as this is a classic bikeshed topic. Those details can be figured out, if needed… (Also, moving things around after the fact would be possible anyway, I guess).

The idea, of course, is not just to have these feature category definitions on paper. The idea is, in a next step, to apply a mechanism to these categories, similar in part to a mixture of the “experimental” and “language import” features in the compiler.

“Language levels” may need to be explicitly declared. Depending on some compiler switch, the use of features in the “advanced” category, or even the use of “standard” features, would have to be manually enabled: Some “language level annotation” would be required at least on a per compilation unit basis, or maybe even (IDK what’s best at the moment) on a package level (like e.g. advanced package foobar).

The use of “language levels” should probably also be some kind of “viral” like “experimental” is: If you directly use dependencies of a higher level than is required to be manually enabled the compiler should yell at you if you don’t annotate your code with the appropriate “language level”.

But “language levels” should not apply directly to expressions unlike “experimental”, or “language imports”. This is to keep them useful, and to avoid micromanagement. It’s a kind of “all or nothing” declaration for larger units of code. That’s because these units are usually used (and read, and hopefully understood) together as a whole. This would also enable more useful metrics, imho. If you have a lot of “advanced” packages (or compilation units?) in your codebase this is a much clearer signal that you’re writing complex code than a few counts of maybe only very loosely scattered “language imports” here and there but possibly everywhere.

For teaching beginners, the compiler would usually be switched to a mode where it only allows “basic” unannotated Scala. The compiler would be able to automatically check if only “basic” features are used, which might be interesting for teachers.

More interestingly, “advanced” Scala could be restricted to only some modules of an application. Organizations could also set limits on the relative number of such “advanced” modules in a code base. At the same time, you could check whether marking things as “advanced” is really necessary, e.g. if a larger module contains only a handful of usages of advanced features (but still needs to be annotated as a whole of course). This could be a signal to refactor the module and get rid of or move the “advanced” parts elsewhere, and free the whole module from the “above average complexity” label.

The point is: Such “language levels” would be actionable because they are quite coarse-grained.

Of course, and as always, adding stuff doesn’t make things simpler, quite the contrary. To keep the balance, I would suggest getting rid of “language imports” at the same time. The features would overlap anyway.

“Language imports” never made sense to me. They’re just an annoying micromanagement chore. They’re not actionable in any way. They don’t provide any insightful metrics. They don’t let people retreat form the usage of certain features. Beginners, in particular, will just hit the yellow light bulb and be cool that the snippet from Stackoverflow is now compiling.

But if you have to tag your entire compilation unit / package / module (?) with something that is considered “problematic” under some aspects, that is a much more serious decision. Maybe introducing “advanced” components requires sign-off from the team lead / some seniors? Maybe your professor won’t accept such a solution? “Language levels” would be a much more powerful tool to enforce some guidelines! Which “language imports” can’t really be, because in practice you can’t micro-manage things in such detail. It’s too much context dependent.

So the TL:DR would be something like this: Replace “language imports” with coarse-grained semantic “language levels”, which are categories of language features. Have the compiler check for proper usage; add tooling that can provide “language level” metrics for a code base.

What needs to be figured out, of course, is which feature goes where. Also, the level of granularity isn’t clear to me yet. My gut is leaning towards whole packages, but I’m not sure. I hope someone has some arguments for a direction here. (It shouldn’t be finer than compilation units, though, I’m pretty sure of that).

This idea doesn’t have a high priority. It’s just an idea for the future. Deciding which features belong in which “subset” of Scala would require a review of the whole language, and that’s a lot of work (and bikesheding). Currently there are more important issues still open. Maybe the right time to think about this proposal in more detail would be when reviewing books for a fully polished Scala 3 at some point in the future?

But I really think this is something worth considering. If you had a fairly reliable switch to confine the power of Scala to just a few dedicated places, and mark those places in a very clear way so that especially automatic tools could flash some warning lights when approaching, this would imho ease the tension between having an adequate language for newcomers and at the same time a mighty tool for power users.

This here needs to end:

Imho it’s possible to end this with this “language levels” proposal. Just draw a clear line in the sand and agree that crossing it requires some kind of pass. Let’s finally stop pulling in two opposite directions.

Scala is for everyone. It’s a big tent!

Without any “pass” (“language level annotation”) you get “standard” Scala (or in case the compiler is in “beginner mode” respectively “basic” Scala). Most people will work with the “standard” code category most of the time, and this should be the “blessed way” communicated to the world. The dedicated experts can take care of the “advanced” components. Beginners can be restricted to “basic” Scala. All this can be easily monitored and enforced.

At the same time, this proposal would, imho, lower the barrier to entry for new features into the language considered “advanced”, as you could more effectively fence off those features from general use. (Something that language imports don’t do!)

Of course, marking things as “advanced” isn’t a bad thing in itself! It just needs to be clearly communicated that this stuff is for more experienced users, and if you commit to too much of this stuff, you may get into trouble at least in terms of finding the appropriate talent to maintain your stuff.

Of course, some amount of “advanced” code is likely to be needed almost everywhere. At least some of your deps will have this mark. But now you can see clearly where the (language) complexity is hiding, and fence off those parts if necessary.

What do you think? Does this make sense? Do you think it would make things better? Could it happen?

Oh, and while we’re at it; not really related, but something that could be done when touching these parts of the language anyway: How about renaming “experimental” to “unstable”? I think that fits the intent much better. But maybe this is just a ticket in the Scala wishlist bug tracker and not a real discussion topic…

4 Likes
2

To be clear: I’m not advocating in any way to make “advanced” features second class.

It’s just to have a clear marker. Libraries could for example equally well advertise that they are (mostly) “basic”, or “advanced”. Such a marker would help in choosing the right tool for the job! (Think filter on Scaladex)

It’s not like “advanced” == “keep away”, “standard” == “blessed”, and “basic” == “just for beginners”. It depends on the context, of course. Having for example a lot of components that are expressed just with the “basic” language could mean that your code “gets things done, straight”. It could be a sign of good engineering. At the same time using quite some “advanced” components in strategic places can mean that you can use some advanced pattern where it makes sense, which will make your life substantially easier. This can be also a sign of good engineering, from someone how knows what they’re doing.

Such a tool would also enable to enforce architectural decisions. Something that is meant to be kept simple could be protected in an appropriate way. Some other parts could be marked in a well visible way as “here you’re going to need some expert knowledge to clear the way”. This could happen already at the architectural level. Which would be actually quite a unique feature for a language.

The other thing is: To identify which parts of the language need to go in which category one would need likely some serious research. I think most “knowledge” in this regard is anecdotic or even just opinion based. So this could be an interesting research topic! Needs quite some data from people who actually teach newcomers. :grin:

3

Ideas like this come up regularly. It’s even been baked into the compiler as “language imports”. It’s been tried.

It doesn’t work. If it did work, it would have had an impact a decade ago, and people would not still be complaining about Scala complexity a decade later

Furthermore, consider how many other languages do this sort of thing. Does Python hide metaclasses under a flag? Does Java block usage of java.reflect unless some special configuration is given? C#? Ruby? Kotlin? Swift? Go? No other language chooses to do it this way. I think that’a pretty strong evidence that this is not a promising avenue for exploration.

There is actually one example I can think of. Elm went all in on restricting usage of their internal/advanced/unsafe APIs, in the interest of making people write “safe” code. It just about killed the language and community. Before the lockdown, Elm was thriving. Now Elm is just about dead.

Apart from the poor track record and social proof, are plenty of technical reasons why this doesn’t work. The largest one being that the whole point of these advanced language features is to make code simpler. Nobody adds language features just for you to make a mess. As a result, prohibiting language features just means the user cannot write simple code in the scenarios the feature is useful. Limiting them to be used under a flag means that everyone just googles the error message and adds the flag as a matter of habit. Nowadays, your IDE even auto-adds the necessary language imports for you. And why not? They just get in the way of writing simple code

People don’t write complicated code unnecessarily. These advanced features are used to avoid writing even messier code: possibly using textual source code generation, post-compile bytecode rewriting, unsafe runtime casting or reflection, or compiler plugins or outright forks. We can see this happening in all other languages that do not support advanced features: take away generics, and people are passing around interface{} and casting like they did in Java 4, or string templating using the canadian aboriginal alphabet. Take away macros, and people are doing textual source code generation or AspectJ-style bytecode rewriting. Is that really an improvement?

Consider Scala 2 macros: yes they are a mess. And yes migration to Scala 3 is a pain. Yes there’s plenty of unhappiness. But what would have happened if Scala 2 macros didn’t exist? That would have killed the entire Scala.js and Scala-Native ecosystems (which doesnt support runtime reflection, and thus relies heavily on macros), large chunks of the com.lihaoyi ecosystem (Cask, Mainargs, Mill, Sourcecode, Fastparse, uPickle), SBT 1.0 (which relies heavily on the .value macro), every serialization library out there, and countless other foundational projects in the Scala ecosystem. IMO there’s a pretty good chance that Scala 2 without macros would never have achieved anything close to what it has today

There’s definitely some amount you can guide people i to doing the “right” thing, but in the end you have to trust people that we’re all adults. Other languages get by just fine without artificial barriers to using advanced features, and Scala has had problems despite having these artificial barriers built in. It seems very unlikely that adding more artificial barriers to using advanced features is the solution to any of the problems the Scala language faces today

15 Likes
4

I agree actually with most of what you’re saying.

I’m in the camp of people who say “a language can’t be powerful enough, just trust me, I know what I’m doing”. [famous last words, I guess :sweat_smile:]

So this proposal is clearly not about restricting things. It’s about marking things.

It’s about making Scala a “better scaling” language. From beginner to advanced user.

It’s not about restricting advanced users! I would hate that… :skull_and_crossbones:

And yes, “language imports” don’t work. That’s what I’m particularly trying to fix here.

2 Likes
5

A few miscellaneous initial thoughts, while I’m digesting this:

I think I was the one who originally suggested the language-level idea, back in the mists of time on the old mailing list, and Martin then fleshed it out into something concrete. I didn’t really perceive it as rejected per se – as you say, it was a bit hard to take action on, and gradually became obsolete over time as the language’s features changed, so it just kind of slowly got ignored. Certainly the details are all wrong now, but I agree that I’ve been thinking a lot about what a revamped set would look like today.

That said, I think Martin’s original analysis was basically right – in particular, the distinction between features relevant for “application-like” code vs “library-like” code. I’m weasel-wording that because it’s not unusual for applications to contain some “library-like” code, designed for reuse, that doesn’t happen to be in libraries.

That maps moderately well to your beginner-advanced-expert lens, in that folks typically start out focused on “applications” – building practical tools – and only later move to “libraries” – building reusable components.

That said, there’s another complication: it’s somewhat common to need to understand a feature a little when starting out, but don’t need to really grok it until later, if even. Variance is the poster child here. IMO most courses spend too much effort on variance too early. Application programmers need to know that the + and - are there for a reason, and vaguely why, but that’s it – it can mostly be explained as “these need to be there in order for this data structure to work the way you expect”, and leave it at that. Until you are building data structures, you can mostly ignore the details of this important but confusing feature.

The moral there is that there’s some subtlety to what it means to “use” a feature. The coarse-grained “levels” idea is interesting, but keep in mind that it breaks down less by feature and more by precisely what you are doing with a feature. That may well be a tractable problem, but it’s a subtle one.

4 Likes
6

Oh, sorry! I don’t remember exactly.

Yes, this makes some sense. But it’s not quite what I had in mind. You can have distinct parts of an application or a library which would fall into one of the proposed categories.

For example a lib can have some “core” component. This would be in a lot of cases an “advanced” codebase. But it could have also some user facing “API” component, which could fall under the “basic” category when it’s usage and understanding only requires “basic” understanding of the language (which would be a clearly demarcated feature set, which is good form the teaching perspective; still the knowledge of just this feature set would enable even beginners to be productive with an API marked as such).

Same goes for applications. They have different parts, like you’ve said. Here being able to differentiate parts on the “language level” axis could also be helpful, in the same way as for library components. It would clearly signal who can or should touch some parts of some codebase.

Yes, that’s what I’ve meant when saying things are “context dependent”.

The “level marks” aren’t there to restrict anything. They are there to mark things. What you’re doing with this information downstream depends on your concrete needs. Tooling can pick them up, but to what effect depends on the circumstances.

It’s coarse-grained so this information doesn’t become just noise (like with the current “language imports”). The provided information is there to give you a high-level overview. Answering the question: Which parts of a codebase are “simple”, which are “average”, and which contain substantial amounts of “magic”? What you do with this info is your choice. You can try to restrict things, sure, but this is a downstream decision (maybe only good for very large and strict organizations). You can also just use this info as guardrails when assigning people to tasks, and nothing more. Or, you can keep learners on track…

The point is in having such clear guardrails in the first place.

Because the goal for me is to enable the addition of more advanced features into the language without the ever repeating “BUT, what about the newcomers? We can’t expose them to even more complexity!”

By drawing a quite clear line between “expertise levels” one could please both camps. Having a clear path to “scaling Scala knowledge”, which is supported by tooling aid, is key here, imho.

Being friendly to newcomers is the only thing that will let your language prosper in the long run. This aspect really can’t be overstated. You really need to care about newcomers. That’s just true. If you overwhelm them, you loose them. Usually forever. That’s not good.

But at the same time keeping people engaged constantly requires to meet their (usually) ever growing demand for more powerful language features. People grow, and so should the language for them. Remember “SCAlable LAnguage”. Otherwise they will move on to where they get the more powerful tools they need to wield to get their now more complex tasks done in reasonable ways.

Marking roughly required expertise levels of code components in a clear way could avoid frustration for people in whichever phase of their personal development they are.

If a beginner touches “advanced” stuff they may expect some pain… If an expert is required to only ever write “standard” Scala maybe it’s time to look for another job where the expertise could be used in a more productive way…

Like I’ve said, it’s about an actionable info or metric, which can be extracted and handled by tooling. It’s not about prescribing people some Scala subset they only may use. It’s about knowing where (language level) complexity is “hidden”. Not more, not less.

Sure, it makes enforcing rules possible. In some cases this is even a good idea! Limiting the available feature set of the language for someone who’s just learning the basics may make things indeed easier for them (especially as it would aid in choosing appropriate libs). At the same time keeping “some people” out of some parts of a codebase may make sense for some organization. But all this is just something that gets enabled downstream. This proposal doesn’t enforce anything like that! It makes only possible to have some guardrails—where needed.

The idea is to please at the same time the “Less power, please!” and the “More power, please!” people, who, like I’ve said, have both very valid needs. Just draw a line in between. Problem solved, once and forever. :grin:

7

Ok new comer here to Scala 3. I have worked with Python, Julia, and the Spark Scala API in the past.

I have been on the ramp up with Scala 3 now for a few months. I had a pretty productive time building and researching trading systems in Python, but it came the time that I needed safety, robustness, along with a flexible high-level statically typed language. I considered quite a few languages, and settled on Scala 3. If Scala 3 did not have some sort of macro, I would either have gone to a another language with macros, or had I stayed in the JVM, then I would have gone to Kotlin (even if I liked Scala better as language, which I do).

So clearly I enjoy Scala’s advanced features albeit being a beginner.

What I also enjoy, and have as as a must in the Scala 3 ecosystem, is tooling. Truthfully, coming from such a mature ecosystem as Python (regardless issues there), our tooling is coming along. A year ago, or two, probably couldn’t have made the decision to use Scala 3 yet because of tooling.

So whatever advanced features there are, they need to be backed by tooling.

I am not going to lie, I am going to continue working with Scala for the foreseeable future in my life long endeavor to build my quant-based hedge fund, and I believe Scala 3 will be pivotal to helping me get there. All that said, I could not in my right mind ever recommend to a client, to venture into Scala for a ordinary green projects for which there are many options out there for, unless if there were special circumstances. And if this is a corporate client, even worse, probably not in a long time to come. However, when I do bring someone, a super capable someone by the way, that person will join me in writing Scala to help me build what I started. I foresee a very small team working with the Scala code base. Should the size of the team ever grow beyond small, I see a shift to another JVM language with less advanced fratures, probably something without Macros, maybe Kotlin. It’s sort of an economic move.

I don’t know if this helps, but at the end there was meant some hidden message in the above:

  1. Scala’s advanced features is what I came for
  2. Only though because tooling became usable with an expectation to keep up with the language in its entirety.
  3. And maybe a little less obvious, I have scoped Scala out with a fallback plan because of potential, language, ecosystem, and even community fragmentation. As such, whatever way this moves forward, please don’t add anything to the language that makes it fragmented. Don’t create multiple languages in one.

Pedro

8

Thanks for stepping in! :+1:

That’s why I’ve chosen neutral abstract category names.

I’ve tried to explain why this doesn’t mean for example “basic” == “beginner”…

Yes, the idea is to avoid severe external fragmentation; by introduction of some useful internal partitioning, which could be used as metric to guide some decisions and a kind of guardrails for some distinct groups. It’s not about splitting the language as such! Quite the contrary. By embracing what happens anyway—but uncontrolled—give things clear structure form the inside. Let people handle that structure according to their needs than.

The language is anyway kind of “diverse” already. Different groups with different requirements need to be pleased. That’s just a reality since forever. So this reality needs to be acknowledged finally instead of trying to make the round peg square.

You need to provide an umbrella, but at the same time give the different groups of people what they want. Otherwise you risk much more severe fragmentation, with camps constantly fighting over “their territory” to keep it “safe” (for them). That’s not sustainable and doesn’t make anybody happy in the end.

Scala was envisioned as scalable language. We should make profit out of that!

How about a “language stack”? :smiley:

9

I think the way to bridge the gap between appealing to both beginners and advanced developers is not this way, but rather:

  1. By having excellent tooling, so that a beginner can encounter advanced code and feel like the tooling is holding their hand and guiding them through it. This means: (a) By default, all error messages should be extremely beginner friendly and offer advice. Advanced developers can turn that off if they want to see more on the screen at a time. (b) A language feature is not considered fully implemented until it has superb IDE support, both in IntelliJ (the Scala plugin and the Community edition are OSS), and in Metals (which pretty much covers all other IDEs).
  2. By having amazing documentation. Scala got way better in this regard a few years ago, but it needs to do even better.
  3. By managing expectations. People should come to Scala not because they mistakenly think it’s more or less like learning any other language, but because they know that once they master it, they can be way more productive than any other language, and that it’s worth the learning curve. (Not to exaggerate the problem – com-lihaoyi or pythonic scala is very approachable and very viable, but one will encounter more advanced things and that shouldn’t be a reason to recoil, rather one should appreciate that they exist for very good reason.)

Years ago, “Scala is too advanced,” “Scala code is scary,” “Scala code looks too unfamiliar,” and so on were believable criticisms. That was a long time ago. That’s what Javascript programmers said. But that was before ES6 became popular. Suddenly the Javascript programmers loved complexity, because they appreciated it.

It was certainly before Typescript, and the unbelievably complex types that people write in it (and in Typescript types can’t even influence behavior, so they’re much less useful than in Scala), became beloved. Now Typescript is tremendously popular, because people appreciate what all that complexity is buying them.

And it was definitely before Rust, and its incredibly noisy and foreign syntax. When Rust came along, it was famous for how bothersome the compiler could be, how hard it was to satisfy the borrow checker – and for how worthwhile that process was. Everybody loved Rust, because everyone knew that the pain was an investment. It was super worthwhile to spend the time it took to get the hang of it, because then you knew your code would be bug-free and fast. Even though for most applications garbage collection is a better solution, the vibe around Rust was always positive, because the ROI was always part of the picture.

Meanwhile, C# keeps on adding new features. So I don’t buy that Scala’s issue is its complexity.

Rust may have a really long learning curve, but everyone knows that. And Typescript may have a really complicated type system, but its documentation is super approachable. The TypeScript Handbook is written in plain, simple English but it’s quite comprehensive and very easy to find things in. And anywhere you look, Rust and Typescript devs are raving about what an amazing language it is.

Scala’s issue is not that it has advanced features. It’s that no one tries very hard to sell it, nobody thinks it’s worth learning, and the people that do encounter it don’t know what to make of it and the friction they hit.

4 Likes
10

Can you elaborate why? What do you find risky when venturing into Scala?

1 Like
11

I believe the culture shock is not so much the language (in fact, written right, it’s about the easiest and most legible there is), but with a large segment of the libraries published in Scala. Scala had the reputation to be simple to learn in the beginning, even though most people then were completely unfamiliar with functional programming. But at that time there were no complicated library stacks and frameworks to cope with.

3 Likes
12

Thanks for the reply!

I agree with most of what you say. But not really the conclusion.

The points 1. and 2. are imho doubtless. That’s the way to go. But these points are orthogonal to this proposal here.

Point 3. is a kind of outcry for better and more coordinated marketing on the one hand side. That’s valid.

But my proposal would actually aid in “managing expectations”. That’s part of the proposal!

The rest is an repetition of arguments I usually bring along: Scala is not “too complex” compared to the “bigger languages”. The issue is in large parts perception and marketing, imho.

BUT, that’s just again trying “to pull the rope” in one direction. This doesn’t work! There are people, with strong and valid arguments on the other side that perceive things differently. My conclusion is, it makes no sense to argue with those people. No “side” will win. We’re just repeating the very same arguments over and over, and don’t move in the end (or move in the direction of some “compromise” which is usually not the best outcome for everybody).

Also your point 3. would equally well “work” for example for C++. But reality shows that this does not work. Nobody invests in a language in the first place “because they know that once they master it, they can be way more productive than any other language”. People invest in a language at first to get shit done, effectively. After you hooked them up the “it will make you a better programmer in the long run” argument may start to work. But it’s not a working sales pitch usually.

The other examples also don’t work in the case of Scala.

Something like Rust is more or less without real competition. The only alternative is C/C++. Sane people than of course prefer to take the pain of learning Rust than the even much bigger pain of learning C/C++.

Same goes for the languages tied to a platform. C#? You need C# if you want to work on M$ stuff (or game dev). No way around. So it makes no difference what the language does. It can be as problematic and mediocre as it wants. A lot people will still use it.

TS also “won” because lack of alternatives and especially a lot of marketing Dollars! (Flow was actually better from the technical standpoint. But that’s moot as always if you have a company like M$ in the back which will spent tens of millions of Dollars to push something into the market.)

Just look how “popular” ObjectiveC was. One of the most horrible languages out there. Nobody sane would use it under normal circumstances. But it was the blessed Apple language. So it was deemed to succeed.

The morale is: You don’t convince people by technical arguments. Never ever. You need to take them by the hand and carefully show around and take the “fear of the unknown” away, and convince them that there is no or only low risk in venturing further.

Also you can’t overwhelm people. Try teaching C++… You get to the core of the problem pretty fast.

So yes you’re right on the one hand side that Scala needs much better marketing (and still some improvements on the ecosystem side).

But the whole discussion is also about convincing the people who (imho rightly!) fear to “spend all the complexity budget” on new, more powerful features. But without convincing this people there won’t be any new and more powerful features…

13

Exactly!

That’s why I proposed some “marker” that would finally work.

The intent behind “language imports” is actually the same, but they don’t work. So we need something that works. So beginners can chose their tools appropriately.

As they grow out of being beginners they can than look around more for other libs. But giving an aid in where to look is important. Especially when you don’t have any overview or clue about the ecosystem!

14

LOL, reddit.

https://www.reddit.com/r/scala/comments/14ii6mg/simpla_replacing_the_concept_of_killer_app_driven

I disagree with some of the details (well, it’s not a good idea to dive head first into the most controversial bikeshed topic while trying to sell an idea), and I don’t think a separate language is a good way.

But at the core someone just proposed more or less the same as I did here: Drawing a line in the sand, to please the “boring industry language, good for beginners, and code monkeys” camp, but still make it a kind of first step on a staircase to a more advanced usage (the infamous Haskalator… :grin:).

Also “funny” to watch the ever same knee-jerk reactions: Some say, “yes sure”, but we don’t need any new “OCaml”, “F#”, “Flix” (OK, nobody mentioned this one by now, even its clearly in the list), and the other side just came along with the default reply “we need to concentrate on the advanced stuff, because this drives current adoption”. BAM!

We really need to end this game of “tug of war”. I can’t watch it any more.

The point is: Even others come up with the (imho obvious) idea of “sublanguages” that stack form “simple, boring” to “the tip of Mount PLT Olympus”. I feel validated… :smiley:

15

I’m not sure chasing being the “boring industry language” is a great idea for Scala. There are already boring industry languages, funded and backed by industrial giants. What will bring people to Scala over Go if Scala has the same power as Go by default? What’s the draw for Scala if it can only do what Java does by default?

The draw of Scala, and what has had it grow and stay alive all these years is the more advanced features it provides making it easier and safer to do things that other languages had trouble doing easily and safely. Scala is where it is because of its advanced features, not in spite of them. I think what’s hurting Scala is that we have a lot of very complex to use libraries, and not as many simple ones.

As an example, li haoyi’s upickle vs circe. Circe is great, and very principled, but using its decoding results are hard for a quick and simple project compared to upickle which just throws if a decoding error happens. Http4s is powerful and rather unopinionated, letting you build your server pretty much any way you want, but actually doing the basics in it is harder than it probably should be.

2 Likes
16

The answer is already in the question. The key word here is “by default”.

You can’t “switch” Go or Java into “Idris mode”, no matter how hard you’re running against some walls.

But in Scala something like that amounts to just importing some “advanced” lib!

Scala scales. Language like Go or Java don’t. Your pretty “quickly” reach a dead end. Form there it just sucks…

But you still need to sell at first to people who “just want get shit done”. Scaring them on first contact means losing them…

I fully agree!

I’m pretty sure if there’s a reason that people sick with Scala it’s because they can do with it even the more complex stuff in a safe and reasonable way. But that’s something where you’re usually “at the end of the road” with one of the other languages! People here may have forgotten, but that’s actually a pretty long way up there. Years…

And how can someone new to the language know that?

They will go and pick something on grounds of the marketing. As the FP marketing is strong currently they will likely end up with for example Circe for JSON.

But think what’s next. After spending a few hours trying to “just read this fucking field from this nested object” the reaction is often: “Scala? Never ever again! Complete trash”. With a little “luck” this will also show up one day later on Reddit and Hacker News: “How I wasted half a day to read a JSON field in Scala” or some crap like that. Boom, we have again the “finest of PR” around which will stick stink for weeks, month, or even years…

We need to break this circle!

17

I’m not sure how language level imports will solve this issue at all. Circe’s decoding is difficult to deal with because decoding anything involves dealing with Either, which is difficult to compound with other monadic types. Locking advanced language features away by default won’t save you from that unless you’re proposing locking parts of the standard library (such as Either) away behind advanced language feature flags.

Moreover, a library using advanced language features doesn’t result in it being hard to use. Again, lihaoyi’s upickle uses a lot of advanced language features under the covers (probably the same set that circe does) but it is still easier to use than circe. I would argue that upickle is also easier to use because of its usage of advanced language features, not in spite of them. The non-advanced alternative would be manual definition of Readers and Writers, which is way harder!

So I’m left puzzled by how locking away advanced features of the language will actually end up helping new users.

edit: I’d like to point out, the scala toolkit is exactly the response to the marketing problem you brought up. Those are tools that are meant to be quick and easy, and they’re being put at the forefront.

4 Likes
18

OK, we’re moving into the implementation direction already. Good. :grin:

First of all this isn’t about “locking anything away”. Maybe I still didn’t explain this sufficiently, but the change on the language side would be a new kind of marker (in the spirit of “language imports”, but very coarse-grained, so no “information noise” or micromanagement).

This marker would be tooling friendly, so you could extract this info automatically. What you do with this marker downstream is your choice. It could for example show up as part of library descriptions, or in you IDE.

But it would have also at the same time a kind of steering property. Part of the idea is that it pushes in the direction of better library design. The core of your library will be in the most cases “advanced” Scala. But this isn’t necessary true for the user-facing API of your lib! (@lihaoyi libs are actually a great example of that!)

Same for application code.

Such a marker would motivate people to encapsulate complex part in dedicated places. The “natural” move would be imho to try to provide surface APIs that are usable with “standard” or even just “basic” Scala.

This would aid the Scala ecosystem. People would be actually motivated to “hide” the complex parts instead of smearing them all over the place, and leaking all kinds of complex stuff into end-user APIs, forcing those users to have advanced knowledge just to be able to use some lib / application API.

So this is in large parts about “culture”. While making at the same time visible to newcomers which parts of the ecosystem support their current level of proficiency. It’s to avoid the “shock moment” for newcomers, who maybe just finished their first FP course, but are confronted right after in the real world with almost the most advanced stuff in the PL field. (People indeed complain online that “you need a PhD in PLT to understand common Scala libs”, even that’s not true of course. But that’s the impression people get often enough form “risking a look”).

Why I think this “culture” thing could work: Well, commiting to something which includes a very visible marker which says that this stuff is “advanced” is sending a quite strong signal. When you do that you know that this will make your code land in some “auto-reject” filter for some applications. You won’t reach everybody anymore.

Also having such a thing as code metric would maybe “calm” some people to not use the most advanced stuff out there just to accomplish something that could be done in a much simpler way. I think it would steer people more towards the “principle of least power”.

I still have to watch it, but here’s something form the cover of a video I have on my list:

def add(a: Int, b: Int): UIO[Int] = ZIO.succeed(a + b)

val list: List[Int] = List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
val sum: UIO[Int] = ZIO.foldLeft(list)(0)(add)

Yes, people write such mind fuck. My proposal could help to prevent that maybe, as you usually would not spend your “advanced module budget” on such crap. You would think twice. That’s part of the goal here.

Sure you need complex stuff—when you need it. That’s why you use Scala in the end!

But you would be inclined to properly encapsulate this stuff. (Something that “language imports” don’t provide). Organizations could, like I’ve said, for example set “limits” on the usage of “advanced” modules. Say only 20% of your components are allowed to be marked as “advanced”. You have then really strong motivation to keep those part in one place, and design stuff around that core in a “simpler” way. This leads also, and even for free, to good architecture.

As a closing note:

Whether Either, or some future Result, belongs into “advanced”, or even “standard” Scala, and isn’t actually part of “basic” Scala is a topic for later. The issue you presented is imho not the well know Result type; the issue is stuff that comes as an after touch form IO actually… But please let’s not open this battlefield right now!

19

But that would mean that if your “core” uses advanced features, then your user facing API will also have to be labeled “advanced” because it depends on the core.

Most of the perceived complexity these days seems to come from “effect systems”, but ironically those systems don’t necessarily depend on complex or advanced language features. Their cores are often implemented in very low level code for performance reasons, they don’t require macros and as long as you stick to a concrete implementation (code against cats.effect.IO instead of cats.effect.Async) you don’t even need higher kinded types or implicits.

20

Hi Martin,

Big fan of yours.

Do know that, as much as this may sound conflicting, I find Scala one of the greatest languages around. I have always looked to justify my use of it. And today, I am having an opportunity to invest in Scala 3. I see Scala 3 as en edge, and in my industry, I will take any edge I can get my hands on.

All of the above eludes to the fact that getting to work with Scala is an investment. But more than that, as it is right now, given the very rich language it is, subpar tooling, and extra ordinary libraries, Scala is truly a big endeavor and investment just to get started, much more than many if not most other established languages.

This is just to get started.

I am certainly not the most capable individual out there, many others do laps around me. I used to think I was really good, then one day I told myself, if you’re there good, go out there and prove it. Then I did, left the day job and never looked back. 5 years on now consulting in DS and DE plus internal endeavors, a BS in mechanical engineering, a masters in computational geophysics, C++ experience, a masters in computer science, and finally my executive MBA, boy Scala is indeed awesome, and damn hard, I mean all things about it. My price tag investing to become proficient and productive, which I am still paying for, is huge!

Now imagine, a small client with a single Python engineer on deck, or large big corporate client with in-house and off shore Java engineers writing that kinda code, and I come along and say, let’s Scala?

Funny enough, out of all of my degrees, the less technical one is the only that really taught me about economy of scales, and Scala just isn’t part of that, all other options considered. Sure, if I get there, and Scala is already in use because someone already made that business decision, great, I will push forward. If not, that hot potato will not land on my hand when some disgruntled Java developer who is not too keen on Scala tells an executive that we did not hit the milestone because I had all of my offshore engineers spend months on picking up Scala.

I do wish Scala were more wide spread, but then it comes down to both a technical and business decision, and as much as I would like to propagate the language forward, I can only afford to be a user right now, and not a language evangelist.

I will continue to invest in this edge, and will offer that edge to other when it’s justifiable to do so. Often, it’s not. Not everyone is building trading systems.

Rooting for the success of Scala, and sorry for being overly direct and honest.

Pedro