Json and a first-class extensible records syntax

Language Design
4

If I understand correctly structural-types use access method by key.
The documentation says that it is for database access.

It is very important to note that most jdbc drivers use access method by index.
I have made a test which illustrates that processing large data by key significantly slower:

by key
  used memory:408000000
  total time:2.458646836
by index
  used memory:17600000
  total time:0.031717541
ratio
  memory:23.18181818181818181818181818181818
  time:77.51694357390442090072493324750491

So I am sure good database support should provide

  • access by index
    It is significantly improve scalability
  • match type transformation
    It is absolutely necessary for good library to be able to transform “column named tuple” to “value named tuple” (Access by name is our the most desirable feature for slick)

I think it can be done by Flyweight pattern

  • compiler should provide values:Product and meta in the case of constant
  • compileer should provide meta in the case of collections and factories
  • library should provide creation of runtime row
  • library should provide mapping between different row types
trait Row extend Product{
   def getMeta():RowMeta
}
object QueryBuilder{
    def executeByQuery[T < QueryColumns, E: QueryToRow[T] ](qc: T)(implicit meta:Meta[E]):List[E]
}
object Main{
  def main():Unit = {
    QueryBuilder.executeByQuery(
       for (c <- coffees) yield (image = c.image)
    ).foreach{r => 
      println(r.image) 
   }
  }
}

Test code

object RowPerformanceTest {
  val arraySize = 100000
  var keyArray: Array[java.util.HashMap[String,BigDecimal]] = _
  var indexArray: Array[Array[BigDecimal]] = _
  var startTime:Long = _
  var startUsedMemory:Long = _
  var endTime: Long = _
  var endUsedMemory: Long = _
  def begin(): Unit ={
    keyArray = Array.ofDim[java.util.HashMap[String,BigDecimal]](arraySize)
    indexArray = Array.ofDim[Array[BigDecimal]](arraySize)
    Runtime.getRuntime.gc()
    startTime = System.nanoTime()
    startUsedMemory = Runtime.getRuntime.totalMemory() - Runtime.getRuntime.freeMemory()
  }
  def end(): Unit = {
    endTime = System.nanoTime()
    Runtime.getRuntime.gc()
    endUsedMemory = Runtime.getRuntime.totalMemory() - Runtime.getRuntime.freeMemory()
  }
  def main(args: Array[String]): Unit = {
    def testByKey(): Unit ={
      var i = 0
      while(i<keyArray.length){
        val map = new util.HashMap[String,BigDecimal]()
        var j = 0
        while(j<40){
          map.put(s"column_$j",BigDecimal(j))
          j=j+1
        }
        keyArray(i)= map
        i=i+1
      }
    }
    def testByIndex(): Unit ={
      var i = 0
      while(i<keyArray.length){
        val array = Array.ofDim[BigDecimal](40)
        var j = 0
        while(j<40){
          array(j)=BigDecimal(j)
          j=j+1
        }
        indexArray(i)= array
        i=i+1
      }
    }
    begin()
    testByKey()
    end()
    begin()
    testByKey()
    end()
    println("by key")
    val byKeyUsedMemory = BigDecimal(endUsedMemory-startUsedMemory)
    val byKeyTotolTime =  BigDecimal(endTime-startTime)/1000000000
    println(s"  used memory:$byKeyUsedMemory")
    println(s"  total time:$byKeyTotolTime")
    begin()
    testByIndex()
    end()
    begin()
    testByIndex()
    end()
    println("by index")
    val byIndexUsedMemory = BigDecimal(endUsedMemory-startUsedMemory)
    val byIndexTotolTime =  BigDecimal(endTime-startTime)/1000000000
    println(s"  used memory:$byIndexUsedMemory")
    println(s"  total time:$byIndexTotolTime")
    println("ratio")
    println(s"  memory:${byKeyUsedMemory/byIndexUsedMemory}")
    println(s"  time:${byKeyTotolTime/byIndexTotolTime}")
  }
}

see also:

1 Like