Tuesday, December 13, 2011

File size in IntelliJ

I'm currently working on a java project and using IntelliJ as my prefered development environment. Yesterday I was generating a jUnit test class with 401 test methods. I realize that a java class with 401 methods in it is an abomination of sorts but it was so much easier to generate it as one file instead of several so I chose to do that nevertheless. Unfortunatly IntelliJ refused to accept my class as code and simply refused to recognise it for what is was. It drove me crazy but the nice folks at jetbrains helped me out. As it turns out there is a setting in idea.properties for this.
#---------------------------------------------------------------------
# Maximum file size (kilobytes) IDE should provide code assistance for.
# The larger file is the slower its editor works and higher overall system memory requirements are
# if code assistance is enabled. Remove this property or set to very large number if you need
# code assistance for any files available regardless their size.
#---------------------------------------------------------------------
idea.max.intellisense.filesize=2500 
So, I simply increased the filesize and that was it.

Friday, December 9, 2011

Speeding up inserts using Linq-To-Sql - Part 2

The latest post in this serie can be found at

Speeding up inserts using Linq-To-Entities - Part 4

Part one is found at http://blog.tanneryd.com/2011/11/speeding-up-inserts-using-linq-to-sql.html.

This version works better if you have nullable columns in your table and it also handles foreign keys. I.e. it will ignore foreign key objects but it requires that you set the foreign key id instead.
    partial class MyDataContext
    {
        partial void OnCreated()
        {
            CommandTimeout = 5 * 60;
        }

        public void BulkInsertAll<T>(IEnumerable<T> entities)
        {
            entities = entities.ToArray();

            string cs = Connection.ConnectionString;
            var conn = new SqlConnection(cs);
            conn.Open();

            Type t = typeof(T);

            var tableAttribute = (TableAttribute)t.GetCustomAttributes(typeof(TableAttribute), false).Single();
            var bulkCopy = new SqlBulkCopy(conn) { DestinationTableName = tableAttribute.Name };

            var properties = t.GetProperties().Where(EventTypeFilter).ToArray();
            var table = new DataTable();
            
            foreach (var property in properties)
            {
                Type propertyType = property.PropertyType;
                if (propertyType.IsGenericType &&
                    propertyType.GetGenericTypeDefinition() == typeof(Nullable<>))
                {
                    propertyType = Nullable.GetUnderlyingType(propertyType);
                }

                table.Columns.Add(new DataColumn(property.Name, propertyType));
            }

            foreach (var entity in entities)
            {
                table.Rows.Add(properties.Select(property => GetPropertyValue(property.GetValue(entity, null))).ToArray());
            }

            bulkCopy.WriteToServer(table);
            conn.Close();
        }

        private bool EventTypeFilter(System.Reflection.PropertyInfo p)
        {
            var attribute = Attribute.GetCustomAttribute(p, typeof (AssociationAttribute)) as AssociationAttribute;

            if (attribute == null) return true;
            if (attribute.IsForeignKey == false) return true; 
            
            return false;
        }

        private object GetPropertyValue(object o)
        {
            if (o == null)
                return DBNull.Value;
            return o;
        }
    }

Monday, November 28, 2011

Speeding up inserts using Linq-To-Sql - Part 1


The latest post in this serie can be found at

Speeding up inserts using Linq-To-Entities - Part 4

I posted a followup on this at http://blog.tanneryd.com/2011/12/speeding-up-inserts-using-linq-to-sql.html.

I use linq-to-sql quite a lot and I really like it. But, if you need to insert huge amounts of data into a table, linq-to-sql is not your best friend. I had a table I needed to initialize with more than a million rows and it took forever, something like 10 hours. I scanned the net for solutions but couldn't find anything I really liked. There were various tips about SqlBulkCopy but the proposed solutions were all really complex and mostly involved generating regular SQL code and I really did not want to do that. I just wanted the linq-to-sql datacontext to have a method like InsertAllOnSubmit that would bulk insert my millions of linq-to-sql entities. So, I wrote my own. The solution I ended up using is the one posted below. It extends the linq-to-sql generated DataContext class and should work just fine for you as well. The CommandTimeout setting isn't necessary but I added it just to show how to set it if you need to.
    partial class MyDataContext
    {
        partial void OnCreated()
        {
            CommandTimeout = 5*60;            
        }

        public void BulkInsertAll<T>(IEnumerable<T> entities)
        {
            string cs = Connection.ConnectionString;
            var conn = new SqlConnection(cs);
            conn.Open();

            Type t = typeof (T);

            var properties = t.GetProperties();
            var table = new DataTable();
            foreach (var property in properties)
            {
                table.Columns.Add(new DataColumn(property.Name, property.PropertyType));
            }

            foreach (var entity in entities)
            {
                table.Rows.Add(properties.Select(property => property.GetValue(entity, null)).ToArray());
            }
            var tableAttribute = (TableAttribute)t.GetCustomAttributes(typeof(TableAttribute), false).Single();
            var bulkCopy = new SqlBulkCopy(conn) { DestinationTableName = tableAttribute.Name };
            bulkCopy.WriteToServer(table);
            conn.Close();
        }
    }

Tuesday, April 12, 2011

The Javasaurus

When Java (the programming language) was born I fell in love with it immediately. For several years java was my preferred development environment. In the early years of the new millenia I happened to be involved in a couple of .NET projects and as the .NET platform matured I became increasingly impressed by it. I became used to having things like LINQ to make my programming life easier. A couple of weeks ago I started working on a project where I'm porting an existing c# system of mine to a java equivalent and it has become clear to me these last weeks that java has been left so far behind by c# that it is almost embarrasing. The simplicity of my c# code is impossible to replicate in java. I try to ease the pain using some of apaches libraries but it doesn't go all the way, not by far. Reading some of the java community responses to LINQ and other language inventions in the .net world makes me wonder if some kind of brainrot started spreading in javaland without me noticing before? Or, is it simply so that anything microsoft does by definition must be bad to a true java follower?

Tuesday, February 8, 2011

Password caching

Everytime I upgrade to a new version of the svn binaries my build system stops working. This would not have to be the case if only I could remember what I'm always doing wrong. I'm using MSBuild projects to build my code so setting up a new development area from scratch means that an msbuild script does all sorts of things, among them, checking out all code using svn.exe. And, this is where it all goes wrong. Since I have password protected my repository the script is challenged with a password request. For some reason the password prompt is kept hidden for me by the msbuild system. Thus, the build script freezes over and times out, I guess, I've never had the patience to wait it out, "kill -KILL" rules! Or, "End Task" as it is called in boring win systems. So, I figured that if I used svn directly from the prompt once and made it cache my password I would be alright. But, unfortunatly I failed to do so, it simply refused to cache anything what-so-ever. I'm sure I did something wrong but it is a little strange since caching passwords is supposed to be default!

Now the solution (one of them anyway) turned out to be TortoiseSVN. Using the tortoise once and checking the save password checkbox resulted in the password actually beeing cached and svn.exe picked it up!