Wednesday, 17 November 2010

Political Action Needed For Net Neutrality In The UK

Government wants to abolish net neutrality in the UK. Not a piece of good news. I really like the way Siva has said it:

There are a couple of different ways to look at this. The romantic way is that we want to have the Internet as the wild frontier for entrepreneurship, and that's a strong case. There's also the liberal free speech argument, which says we want the Internet to be a level playing field so a variety of voices can enter the public sphere. That's a fairly strong argument. But then you've got the economic argument, which is those of us who write checks every month to these companies, we want to be able to know that we are getting decent service for what we're paying. If my broadband company next week starts dialing down my Skype speed so Skype doesn't work as well for me, I might not even know it or notice it for a long time, until Skype starts frustrating me, and out of frustration, I'm just going to pick up my old phone and dial India the old-fashioned way and just pay for it because I know the call's going to go through. That's the sort of frustration and opacity we might start seeing on the Internet. So it is a service question, a competition question, an economic development question, a consumer question.

I expect to see a political campaign started against this motion, similar to NO2ID. This I will wholeheartedly support.

Saturday, 7 August 2010

Hamcrest 1.1, AllOf, DSL, SuppressWarnings And Maven Central

Hi,

I've never used Hamcrest before; today is my first day with it. And I have already had a lot of "fun" which I would like to share.

My experiences today have been centred around the AllOf matcher. Or rather around this version shipped with Hamcrest 1.1. This is the newest version you can find on Maven central. It also seems to be built into the newest JUnit jar version 4.8.1.

Imagine that in your test you've got variable a of type A and you want to check it against three matchers m1, m2 and m3 of type Matcher<A>. It was quite natural for me to write it like this:

import static org.hamcrest.core.AllOf.allOf;
import static org.junit.Assert.assertThat;
...

@Test
public void testA() {
    ...
    assertThat(a, allOf(m1, m2, m3)); // here you get a warning
}

That works and looks nice. But you get a compiler warning: Type safety : A generic array of Matcher is created for a varargs parameter. I have found this blog post to be very helpful in explaining the nature of this warning. The quickest solution is to write:

@Test
@SuppressWarnings("unchecked")
public void testA() {
    ...
    assertThat(a, allOf(m1, m2, m3));
}

However it would be nice if we could preserve full type checking in the test code. It took me a while to find an alternative solution:

@Test
public void testA() {
    ...
    List<Matcher<? extends A>> matchers
            = new LinkedList<Matcher<? extends A>>();
    matchers.add(m1);
    matchers.add(m2);
    matchers.add(m3);

    assertThat(a, allOf(matchers));
}

This is fully type checked. Unfortunately it is a bit more long-winded and we looses the DSL feeling. On a side note this code can not be shortened to just:

@Test
public void testA() {
    ...
    // doesn't compile
    List<Matcher<? extends A>> matchers = Arrays.asList(m1, m2, m3);

    assertThat(a, allOf(matchers));
}

This sample deceptively looks as legit as the one above it. However it fails to compile because of a Type mismatch: cannot convert from List<Matcher<A>> to List<Matcher<? extends A>>.

I've had to discovere all this the hard way because in Hamcrest 1.1 AllOf.allOf is defined as:

public static Matcher allOf(
        Matcher<? extends T>... matchers) {
    return allOf(Arrays.asList(matchers));
}

public static Matcher allOf(
        Iterable<Matcher<? extends T>> matchers) {
    return new AllOf(matchers);
}


My day would have been saved if I had switched to Hamcrest 1.2. Nat Pryce has fixed the issue in revision 258. We get explicit overloads of allOf method for up to 6 matchers in one go. This makes my very first code sample compile and run without any warnings.

It is also interesting to note that shortly before this commit another issue had been corrected in Hamcrest codebase: revision 194. Matcher<? extends A> has been replaced with Matcher<? super A>. This is most reasonable: indeed we can apply a Matcher<Object> to test a String. On the other hand it is not very useful to apply a Matcher<SomeClassExtendingA> when our object is known to be of class A.

1.2 release of Hamcrest contains both fixes. Unfortunately it is only available on the project website but not on Maven central. Having spent quite a bit of time on the investigation what do I do now? I see three options:
  • use the long-winded version of code
  • wait for Hamcrest 1.2 to be uploaded to Maven central
  • copy-paste AllOf into my own code base and backport these two fixes

I have chosen the last option. This is my temporary fix. It makes my test code look nice right now and I can easily take out my hack later when a newer Hamcrest appears on Maven central.

Speaking of which: to use an updated version of Hamcrest I will need a version of JUnit which doesn't package hamcrest classes inside its own jar. Such a version exists, it is called junit-dep. Unfortunately the latest version Maven central currently has is only 4.5. Does anybody know of any way to cause a newer version of junit-dep to be uploaded? As far as I can tell there is just one person behind JUnit, Kent Beck. Is this right? Is there any chance to accomplish this with or without his help?

Sunday, 2 August 2009

shoal/jxta - any good?

Hi,

looking for re-usable code to drive a clustered java application.

Shoal promises
  • peers discovery

  • group membership

  • group messaging

  • small replicated cache
About what I need. Used by Glassfish and seems to have other uses too. There is recent activity on the project. GPL not ASL but I it's an in-house project only.

Deal? Invest more effort? Hold on! Here's their list of open bugs
  • (36) Messages received not in same order as when sent

  • (61) When members join the group concurrently, join notifications of some members are often duplicated or missed

  • (74) potential to miss FAILURE_NOTIFICATION when multiple instances killed at same time

  • (83) When group leader failed, any member couldn't receive FailureRecovery notification
Now, the've got versions 1.0 and 1.1.
1.1 is said to be a "work in progress".
So is 1.0 okay? I don't know

Another oddity: a CVS commit message seems to imply (83) has been fixed.
But it's still open in the issue tracker



Jxta - the foundation of Shoal - is another unknown.

When I troubleshoot late at night I prefer to have an intimate knowledge of my whole stack.
Jxta is another big (?) thing to learn. Thankfully sources are available but still..



Does it look easier just to write my clustering code from scratch?
Implement proper Paxos?
Be less dependent on other people's bugs?



P.S. I've had a look at Geronimo clustering as well. They allow Cache servers to be in separate jvm-s. Hmm.. Somehow my intuition doesn't immediately suggest integrating that code is an easy route either..

P.P.S. Zookeeper was another obvious candidate. What stops me here is that Zookeper seems to have a very hard dependency on disk. Every change is persisted. Node recovery is done by reading the transaction log. It's a mismatch for my goals. I need something blazingly fast living completely in RAM

Friday, 12 December 2008

eclipse -clean

Hi! I've got an irresistable urge to share this.

eclipse -clean

cleans eclipse caches and can make a very stubborn build error go away.

I've had a "resource already exists on disk" in a m2eclipse driven project. Deleting the file manaully/refresh/clean/close-open-project did not help. workspace/.metadata/.log contained a lot of ClassCastException-s.

But closing Eclipse and re-running it as "eclipse -clean" did help.

Friday, 30 May 2008

SLF4J - Beyond Log4j

Just starting to work on a new project I've found a very encouraging surprise.
This project uses SLF4J/Logback!

Log4J is a good old fried. However it may benefit a bit of rework.
SL4J is a fresh wind. Same spirit slimmer code.

I really feel it can blow Log4J out of the water.
If it chooses MIT license me thinks.

Speaking of license choices:

Ceki> Do we want Big Iron (insert your least favorite IT company here)
Ceki> to legally pirate an open source project, say logback?

Let's assume SLF4J gains in popularity.
Then it will likely be perceived as same in kind to Log4J.
However then Big Co-s would still hold on to the conveniently licensed alternative wouldn't they?

I don't see how Log4J can be replaced this way.
IMO SLF4J/Logback need MIT/ASL.

Thursday, 17 April 2008

ConcurrentLinkedQueue On A Diet

News are coming very slowly on me I know. This overview of atomic variables from IBM website is dated year 2004.

The article implies that in the case when contention is high (multiple threads compete for access to the same data structures) we can save a lot of CPU cycles that the JVM would have otherwise spent on scheduling threads by using atomic variables instead of traditional synchronized locking.

As I understand Java memory model each read/write to an atomic variable (as well as to a volatile one) introduces a memory barrier forcing the CPU-s to discard read caches/to flush write ones.

Following this line of reasoning I have concluded that it would be beneficial not only to synchronize access to highly contended data structures with atomic variables but also to do as little synchronization as at all possible.

Here's my mental experiment. Suppose we would like to optimize one particular use case. A number of threads put tasks into a queue. A single worker thread takes tasks from the queue and executes them. Apparently we need only one atomic variable to perform this task reliably. Here's the code

package org.apache.atagunov.mr.queue;

public class MrLinked<T>
{
    /** Managed exclusively by {@link MrQueue} */
    T first, next;
}

package org.apache.atagunov.mr.queue;

public interface MrInternalIterator<T>
{
    void process(T t);
}

package org.apache.atagunov.mr.queue;

import java.util.concurrent.atomic.AtomicReference;

import org.apache.log4j.Logger;

public class MrQueue<T extends MrLinked<T>>
{
    private final AtomicReference<T> last = new AtomicReference<T>();
    
    /**
     * Add <tt>item</tt> to this queue
     * May be called from multiple threads.
     * @return <tt>true</tt> if we have gone from empty to non-empty state
     */
    public boolean add(final T item)
    {
        item.next = null;
        T oldLast;
        
        do
        {
            /* suppose queue is empty; then item will be the only element */
            item.first = item;
            do
            {
                if (last.compareAndSet(null, item)) {
                    return true;
                }
                
                /* okay, queue wasn't empty during compareAndSet */
                oldLast = last.get();
            }
            /* may have gone null by now */
            while (oldLast == null);
            
            /* suppose the queue doesn't change until next compareAndSet */
            oldLast.next = item;
            
            /* all items in the queue have same first */
            item.first = oldLast.first;
        }
        while (!last.compareAndSet(oldLast, item));
        
        /* if we got here this means we have added item to non-empty queue */
        return false;
    }
    
    private final Logger logger = Logger.getLogger(MrQueue.class);
    
    /**
     * Run <tt>ii</tt> for every item in the queue then clear the queue.
     * Even if <tt>ii</tt> throws an exception we consider the item as processed
     * Run from one thread at a time please!
     * @return number of orders executed
     */
    public int processAndClear(final MrInternalIterator<T> ii)
    {
        T stopHere = last.get();
        if (stopHere == null) {
            /* don't expect this in context of MrUniverse */
            return 0;
        }
        
        int count = 0;        
        T t = stopHere.first;
        
        for(;;)
        {
            for(;;)
            {
                if (t == null) {
                    logger.fatal("encountered null, queue discarded");
                    /* sort of recovery - discard queue */
                    last.set(null);
                    return count;
                }
                
                doProcess(t, ii);
                count++;
                
                if (t == stopHere) {
                    break;
                }
                t = t.next;
            }
            
            if (last.compareAndSet(stopHere, null))
            {
                /* done! */
                return count;
            }
            
            /* somebody must have added an item to the queue */
            stopHere = last.get();
            
            /* only read .next now after get() which has been a synch point */
            t = t.next;
        }
    }
    
    private final void doProcess(final T t, final MrInternalIterator<T> ii)
    {
        try
        {
            ii.process(t);
        }
        catch (Error e)
        {
            /* sort of recoverty - discard queue */
            logger.fatal("Iterator thrown " + e.getClass().getName()
                    + ", queue discarded");
            last.set(null);
            
            /* let it go through killing the thread maybe */
            throw e;
        }
        catch (Throwable thr)
        {
            /* consider item as processed; more handling may be added later,
             * at least better logging */
            logger.error("Problem processing queued item", thr);
        }        
    }
}


Only one atomic variable per queue! Contrast this with ConcurrentLinkedQueue which uses atomic access for each "next" pointer in its backing linked list implementation. This is our win for coding specialized solution for a specific use case.

I was also so concerned about performance that I've decided to unite backing linked list element implementation with actual data items. Sort of what we used to do in good old C days coding lists by hand. The actual class that needs to be stored in MrQueue has to extend MrQueue<its-own-type>. The gain here is that GC has to take care only about one object not about two.

I have tested this little bit of code as part of a bigger application on a big multi-processor box. The tests have run fine and it appears that the code actually works as designed.

Friday, 21 March 2008

Singleton Free JRE?

Briefing

IoC rocks.
Singletons are the opposite of IoC.

TDD/unit testing/mock objects rock.
Singletons subvert unit testing.

[Objective No. 1] Modularity

I'd like my application components (osgi bundles, ear-s, war-s or otherwise) to be truly separate from the rest of the system.

So that a web container could provide its own implementation of file system, console, threading API to my webapp. And a different implementation to another guy's webapp.

[Objective No. 2] Unit Testability

I'd like to unit test every bit of code I write. For every unit I test I need to mock all it's external dependencies.

I would also like to unit test 3rd party modules if need be.

What Are The Evil Singletons?

These are the complex application entities that can not be mocked for testing
  • constructors, static methods, static variables (Java)
  • functions (C)
What Singletons Are Ok?

Singletons so trivial that dependency on them does not violate modularity and we never need to mock them for testing either
  • constants (Boolean.TRUE, Enum values)
  • constructors of value types (java.lang.Integer(int))
State Of Art

Spring, PicoContainer, Felix (OSGI), Tuscany(SCA) all give us the blessing of IoC. J2EE also does a good job of abstracting inter-module and container services dependencies via interfaces. A small fault is that J2EE components still have to depend on certain singletons.

J2SE is more troublesome. These are the prime examples of what I'd like to mock for testing but can not
  • System.currentTimeMillis()
  • new FileOutputStream()
  • new File("a").mkdir
  • all of Swing
Dreams

I've long dreamed of writing alternative Java libraries. I don't have any exact design. Just an intuition of strict API/Impl separation + Spring/OSGI/Pico to tie modules together.

Another dream is to have this supported at programming language level. Each application consists of module instances. Each module declares its dependencies. The module's instantiator provides them.

Top level module instantiates and connects subordinate modules.
Subordinate modules do it recursively.

Only the top level module has access to true singletons. True singletons are operating system services like IO console thread management.

Only the top level module can provide the true singletons to subordinate modules. Subordinate modules have no way to know if they have been provided with real system services or fake ones.

Perhaps some young and daring programming language has a chance to get this right?