Pre-sip scaladoc: search by type signature

Basically Hoogle-like functionality, yeah. I find myself wanting this all the time. Even within a single type, I often want to narrow down the members by signature…

6 Likes

There’s something similar here: http://scala-search.org/.

I’d say this does not require a SIP, but rather someone that is willing to analyze the existing solutions and has the time to open a PR to scala/scala with a solid implementation.

I personally think that Scaladoc should be made independent from the compiler and be implemented in terms of tools like Scalameta (reading type information from the Semanticdb). So, if more work is to happen in this area, I would encourage contributors to try to implement it on their own :slight_smile:

4 Likes

One annoyance one might run into, I suspect, is subtyping vs unification. Hoogle can use unification, and I’d guess it needs to (though I haven’t read the papers on which it’s based), but unification and subtyping don’t play nice with each other.
IIUC Dotty implements a constraint solver for subtyping (which I think is more robust), while Scalac does something else. Maybe you can stick to something Hoogle-like and pretend subtyping doesn’t exist, but then searching for List[A].?(A => List[B]): List[B] wouldn’t find List[A].?(A => Traversable[B]): List[B].

Martin also mentioned that Dotty’s constraint solver has some known incompleteness to avoid exponential slowdown, I don’t know if search should have the same tradeoffs; but I expect a first implementation should ignore such questions, otherwise this becomes a topic for a whole PhD.

There is a related project built with Scalameta: Metadoc, a code browser with some IDE-like features. I think it could be useful for building a semantic search tool proposed here.

3 Likes

Thanks for all the replies - learned about some really cool tolls out there. I particularly like how Hoogle allows you to enter the signatures … that would make a lot of sense for scala too Int => Codec[Bool] for example.

In terms of the implementation, my understanding is that hte current search functionality in scaladoc API is just some javascript in the generated html pages? Is that not correct?

I don’t believe there is any need to do any parsing or syntax analysis.

There are lots of great ideas here, but aren’t they a bit of overkill for something that you could probably implement by tweaking the js regex for the search functionality (assuming that’s how the search box works?)?

I don’t know how Scaladoc works internally. I suggest you have a look, if you find a way of implementing this I think it’s worth a PR.

Some things might be doable through plain regexes. What Hoogle does certainly not - you can write Int => Int and find the identity function (since it’s more general but can do what you want). But I agree a simple search would be better than nothing!

Hey all. Sorry to revive this old thread from simpler times.

I really feel this is something that could be very useful as well. Given that it has been 3 years, does anyone know of new developments along these lines?

(I’ll be coming back to Scala after some time in Haskell/PureScript land, and I’d certainly do what I can to make this a reality).

6 Likes

Scalameta and/or Semanticdb might be helpful here (and didn’t exist until recent years)

3 Likes

This is for Kotlin, but written in Scala by some of the same folks working on scaladoc for Scala 3:

Therefore, having Hoogle for Kotlin is like proof of concept for having a similar tool in Scala3 world.

3 Likes

We’re in the process of analyzing the amount of work needed to implement/adapt Inkuire for Scala 3.

9 Likes

Really awesome to hear about Inkuire being well placed to tackle this, @griggt and @KacperFKorban .

A few questions - looking over the issue tracker, it seems there isn’t a discussion of the mentioned evaluation of work needed to support Scala 3. Did I miss it, or would that be helpful?

I did see Migrate to Scala3 · Issue #169 · VirtusLab/Inkuire · GitHub, however, I assume that’s referring to migrating the codebase to Scala 3. If that’s the case, which would be better off to tackle first?

I also wonder to what extend non Scala-3 libraries might be supported. Down the road, it might be useful to have a way to crawl through JVM bytecode to extract signatures, for those interested in it (I realize not everyone will be, as not everyone is using Scala for the JVM).

Hi @bbarker the evaluation was within VirtusLab/scaladoc team. We’re currently working on determining whether Inkuire is able to work with data from Scala3 sources. We should have some news of MVP soon.

2 Likes

Hi! As promised, I’m back with an update. Within the Virtuslab team, we’ve been working on adapting Inkuire to work with Scala 3 and scaladoc. I’m happy to say that we have a PoC ready. You can try it out here.
And here is how it looks:

Some info about how to use it:

  • Inkuire works in the scaladoc searchbar- it triggers for queries with signature-like structure
  • Accepted syntax is that of a curried function in Scala3

Example queries:

  • BigDecimal => Byte - toByte
  • Set[Long] => Long => Boolean - contains
  • Seq[Int] => (Int => Long) => Seq[Long] - map
  • [A, B] => A => B => A - const

This is as I mentioned before a PoC, so there are some things that don’t work or are buggy:

  • extension methods – should be easy enough, just not worth it for now, since stdlib doesn’t have a lot of them
  • java classes – no tasty files for them, so will be more tricky :thinking:
  • implicit things – if You notice that there are no results for e.g. String => Int, that is why
  • Type lambdas and HKTs – Can be a bit buggy or inaccurate

If You have any suggestions, improvements or find any bugs, issues on Inkuire repo are very much appreciated :smiley:

Let us know what You think :grinning_face_with_smiling_eyes:

32 Likes

This is sick. By which I mean really cool. If we can get this integrated into Metals one day I’ll be a happier person. Thank you for your work!

2 Likes

We were already thinking about it and should be doable. I am thinking of maybe adding a link to hover that could open up the docs in webview? It would also work via command.

4 Likes

Amazing.

Slight nitpick:

[A, B] => A => B => A

Isn’t the first arrow wrong? Shouldn’t it either be =>> like Scala 3 polymorphic functions or omitted the way Scala 2 prints method types?

[A, B] =>> A => B => A would be a type lambda taking two types and returning a function type A => B => A. [A, B] => A => B => A is a polymorphic function taking two type arguments.

2 Likes

I’ve been wanting this for oh-so-long!

1 Like

TL;DR 1.0.0-M1 of Inkuire is in new scaladoc. You can test it here

Hi, I haven’t posted any updates for a while.
I’m glad to say that version 1.0.0-M1 of Inkuire is available in new scaladoc.
The engine itself hasn’t changed that much, but it should be easier to use now. And now that most technical issues are resolved, it will be way easier to develop new features (aka removing hacks) :slight_smile:

A bit about how it can be used: Searching functions with Inkuire is done by inputting the signature string in scaladoc searchbar (shortcut /). Any string that looks like a signature (contains =>) will be searched using Inkuire, otherwise the query will be searched with normal scaladoc search engine.

I made some changes to the parser since the last post. Most notably I saw that a lot of queries were in the form of List[A] => (A => B) => List[B] and since the engine required for every type variable to be explicitly decalared it silenctly failed on resolving types :confused: So now any type that is in the form of a single letter or a single letter with a digit are automatically resolved as type variables. (Seems like a good enough heuristic for now)

Since no one actually reads long descriptions, here are some examples of searches with intended functions:

  • Seq[Int] => (Int => Long) => Seq[Long]IterableOps.map
  • (A, B) => AProduct2._1
  • Set[Long] => Long => BooleanSetOps.contains
  • BigDecimal => ByteScalaNumericAnyConversions.toByte
  • Int => Long => IntFunction.const
  • String => Int => CharStringOps.apply

Gif for attention:

15 Likes