There ares simple use cases like counters which are awkward to handle concurrently using namespaces or actors. I'm thinking about creating a new class which works a bit like java.util.concurrent.atomic, but is more of a generic Obj wrapper.

I propose a new class called sys::Atomic which used to atomically manage a single reference to an immutable object:

final const class Atomic
{
  // val itself must be immutable
  static Atomic make(Obj val)

  // get the value
  Obj val()

  // Apply the function holding a lock to update val reference
  Obj set(|Obj->Obj| f)

  // only available if constructed with Int 
  Int increment()
  Int decrement()    
}

The constructor would throw an exception if the value itself wasn't immutable, so really what atomic is providing is just a way to synchronously update the value. It provides a volatile/monitor combo which fits into Fan's const/immutable model.

In the case of a sys::Int this would map to a AtomicLong on the JVM under the covers. We could go hog while and create different subclasses/another pod - but I think Fan's semi-dynamic style is just one class that works with Obj (even if some methods throw UnsupportedErr).

Examples:

const static Atomic counter := Atomic(0)  
next := counter.increment

const static Atomic lastSearchTerm := Atomic("")
lastSearchTerm.update { "search term" }

I think this should be called AtomicInteger, if you really want to do it, because increment and decrement are clearly specific to integers and do not apply to anything else (thus, the design above is an instance of a superclass acquiring properties of subclasses, only in this case there are no subclasses).

You could generalize it by allowing user-defined behavior on set:

final const class Atomic
{
  // initial must be immutable
  static Atomic make(Obj initial)

  // get the value
  Obj val()

  // Apply the function holding a lock to update val reference
  Obj set(|Obj->Obj| f)

  // Use the updater to update the value
  Void update(|Obj->Obj| updater)
}

In this way, a user-supplied function updater would be responsible for taking the old value and producing a new value.

myVal.update(Incrementer)

myVal.update(Decrementer)

A subclass called AtomicInteger would provide shortcuts for these called increment and decrement.

The fundamental problem here is that the Fan type system has no way to express the notion of an immutable function (or maybe I am wrong, I haven't tried).

Finally, with STM, all of this becomes a moot point, because you could write code similar to the following:

shared Int myVal
...

atomic {
   ++myVal
}

I haven't thought it through, but it's possible STM can be done with compiler plug-ins, facets, and a backdoor (Unsafe), in which case the above would look like:

@shared Int myVal
...

atomic <[
   ++myVal
]>

The real problem is that to be useful, you need to be able to "send" such shared data to other threads with no overhead. Which means you need to pretend they are immutable. Which makes it uglier:

@shared=Int#
Shared myVal
...

atomic <[
   ++myVal
]>

but (potentially) still doable.

The problem with STM is that you still can't use a static field to store a counter. There has to be some notion of a class like Unsafe we can use to wrap a mutable reference. Perhaps Ref would be a better name. This really isn't unlike what Rich did with Clojure state to separate identity (the reference/Atomic instance) from the state (the current value).

So I think this class is really the first step to implement STM (if decide to do that, I don't want to tackle for 1.0). So I propose we rename the class Ref instead of Atomic. In our case setting a Ref is assumed to be atomic, and in the future could be done in the context of a transaction.

I really dislike polluting the namespace with a special subclass for Int. But I will go with what all you guys want. Here are the options:

1. Single Ref class which Int specific counter methods which throw UnsupportedErr if used with other value types (my vote)
2. Single Ref class and no special Int support, you update the counter using set, no optimization using AtomicLong - simple, but probably slightly less performant
3. Subclass Ref with special IntRef class which can be optimized to use AtomicLong

I like Brian's proposal. Though I think it would be cleaner w/o the inc/dec methods. I think just using set isn't that bad:

const static Atomic counter := Atomic(0)  
next := counter.set |v| { v++ }

While we're on this topic, we also need a way to easily create concurrent Maps and Lists. Not sure what that looks like - but its way to much work at the moment given how common it is.

I agree with not polluting the namespace, so I vote for #2:

Single Ref class and no special Int support, you update the counter using set, no optimization using AtomicLong - simple, but probably slightly less performant

What if we made atomic a keyword and used it as a field modifier, and let the compiler do all the magic? Then you just have

atomic static Int counter := 0

next := counter++

While we're on this topic, we also need a way to easily create concurrent Maps and Lists. Not sure what that looks like - but its way to much work at the moment given how common it is.

I am not sure we should tackle that in addition to namespaces (which is a concurrent map) and actors (which wrap collections with higher granularity). Although if you have a map/list which doesn't mutate very often, this feature could be used quite efficiently:

static const Ref list := Ref(Str[,])

list.set |Str[] x->Str[]| { x.rw.add("new value").toImmutable }

What if we made atomic a keyword and used it as a field modifier, and let the compiler do all the magic? Then you just have

That doesn't work, you have to create an API to ensure that all updates happen within a function passed in (which can be wrapped by a lock).

The problem with STM is that you still can't use a static field to store a counter

Why not?

Perhaps Ref would be a better name.

The only problem with approach is that it forces casting. Clojure doesn't have that problem since it's a dynamic language.

With a keyword, you can retain type:

shared Int myVal
...
++myVal // OK

but with wrapping, you cannot:

Ref myVal
...
++myVal // Not OK

Now with a facet and compiler plug-in, at least the plug-in can know the type, but it does mean a lot more implementation work:

@shared=Int#
Ref myVal
...
atomic <[
   ++myVal // OK
]>

Now if we want a stepping stone to native STM, then perhaps a Ref class with get/set. Then shared keyword & the associated type becomes essentially syntax sugar, with all operations desugared to operations on Refs.

static shared Int myVal
...
atomic {
   ++myVal
}

// Desugared to:
Ref myVal
...
atomic {
   myVal.set(1 + myVal.get())
}

Now as to the options:

Single Ref class which Int specific counter methods which throw UnsupportedErr if used with other value types (my vote)

Don't like this because it's a design smell (superclass acquiring traits of more specific subclasses).

Single Ref class and no special Int support, you update the counter using set, no optimization using AtomicLong - simple, but probably slightly less performant

This won't provide atomic increment/decrement, which requires atomicity of a get and set operation, not just atomicity of a set operation.

Subclass Ref with special IntRef class which can be optimized to use AtomicLong

Of these three options, I think this one is most compatible with future native STM. Ref can have get and set, and IntRef can have whatever it wants (increment / decrement), necessarily implemented with "magic" since get and set primitives are not sufficient to provide atomic get + set. But with native STM, the subclass IntRef could be implemented in straightforward fashion:

Int increment() {
   atomic {
      Int old = get()

      set(1 + old)

      return old
   }
}

which would support legacy code (though future code would simply use atomic { ++myVal }).

EDIT: I see set accepts a closure, so my comments about (2) are incorrect.

While we're on this topic, we also need a way to easily create concurrent Maps and Lists. Not sure what that looks like - but its way to much work at the moment given how common it is.

Going down this road leads you to Java. Rather than special purpose concurrent elements, it makes much more sense to allowed shared state, but only when wrapped in atomic blocks. This eliminates deadlocks and ensures that if code is correctly designed, it will compose correctly -- which is the #1 problem with traditional lock-based concurrency.

I like the shared modifier with atomic blocks as suggested by JohnDG. One thing I'm wondering is whether shared could also be used for non-static fields and thus eliminate the need for Context with actors.

The only problem with approach is that it forces casting. Clojure doesn't have that problem since it's a dynamic language.

I would argue that you hardly ever cast in Fan since most of the time the compiler automatically casts for you. So I think it is OK to use Obj based solutions.

My goal is here is to solve a specific problem using an API which does not require adding new language features. Although I think it is good think about STM features, especially to enable DSL plugins for the more adventurous. But to really explore adding new keywords needs to be a full fledged effort to add STM to Fan - I am not ready for that level of commitment, I merely want to solve the counter problem with an API.

John I saw your edit, not sure we are on the same page, but the simple Ref proposal I have would support counters just fine:

static const Ref counter := Ref(0)
static Int count() { counter |Int x->Int| { x++ } }

The set is always done in the context of a function using this Java implementation:

Object get() { return this.val; }

synchronized Object set(Func func)
{
  return this.val = func.call1(this.val);
}

volatile Object val;

I would argue that you hardly ever cast in Fan since most of the time the compiler automatically casts for you. So I think it is OK to use Obj based solutions.

The compiler won't cast if you try to use a method on an Obj. Which means you would have to manually cast.

John I saw your edit, not sure we are on the same page, but the simple Ref proposal I have would support counters just fine:

Yes, I see that now (I missed the signature of set originally).

Note that as far as I can see, this proposal introduces deadlocks into Fan (since the set closures may access other static Refs), so I'm not in favor of it.

If you want to solve this with an API, I'd suggest using AtomicInteger and not trying to generalize it, because I don't think you can generalize it in a safe way without introducing new language features or at least using a compiler plug-in.

But to really explore adding new keywords needs to be a full fledged effort to add STM to Fan

By the way, a naive STM can be done in 500 lines of code or less. Which I would likely volunteer to write (at least for the Java side). :)

Note that as far as I can see, this proposal introduces deadlocks into Fan (since the set closures may access other static Refs), so I'm not in favor of it.

I agree, but you can introduce deadlocks with actors too. But I think these models force you to think about concurrency in a much more structured way, so even though they aren't perfect, they are a huge leap over Java styled monitors.

STM can detect and avoid potential deadlocks, but it introduces its own sneaky problems. I have to catch transaction commit failures and retry them, or if the implementation retries for me I have to be aware that my code my might execute multiple times. Is that the sort of implementation you are thinking of?

STM can detect and avoid potential deadlocks, but it introduces its own sneaky problems. I have to catch transaction commit failures and retry them, or if the implementation retries for me I have to be aware that my code my might execute multiple times. Is that the sort of implementation you are thinking of?

No, that's not a necessary part of STM, just a common method of implementation that generally offers high performance.

For a first-pass implementation, I wouldn't need retries: with composite Refs, unique comparable identifiers for each leaf Ref, and an embedded reservable lock in each Ref, I could do in-order, optimistic locking for all Refs occurring in the atomic block. This would offer correct behavior but suffer performance problems when used with large shared data structures like maps.

For a production implementation, I'd want to forbid message passing in atomic blocks, and with this restriction, and for the Fan language, I'm fairly sure I could do retries without the developer having to structure their atomic code in any particular way.

If this gets fixed as a library, I'd want to see two classes - AtomicRef and AtomicInt.

I'd also like to see the set() method separate to the closure accepting alter() method.

In the bigger picture, I like the concept of annotating the variable with atomic or shared, although that may not actually be practical in reality.

In the bigger picture, I like the concept of annotating the variable with atomic or shared, although that may not actually be practical in reality.

Actually, STM offers the highest theoretical performance, the safest, most composable semantics, and looks to receive support in commodity hardware within the next few years. Of course, the feature could be added in Fan version 10.x, or whatever, so I don't mind the current proposal (I just don't intend to use it).

Going down this road leads you to Java. Rather than special purpose concurrent elements, it makes much more sense to allowed shared state, but only when wrapped in atomic blocks.

I didnt mean to imply special types - I agree everything should be handled with a single construct if possible.

list.set |Str[] x->Str[]| { x.rw.add("new value").toImmutable }

Yeah, I thought about that too.

Based on this discussion, I kind of feeling like anything we do now is a temporary hack. I don't feel this is absolutely required right now - the work arounds are to use the namespace, actors, or FFI. So I don't feel a pressing need to address this issue immediately, since it really belongs as piece of the larger STM problem (maybe).

So I withdraw the proposal. Thanks for the feedback.