Internship Report

On

Dynamic Query Processing using C#

Prepared by,

Jerin J. John

Year: 2015-2016

K. J. Somaiya College of Engineering Vidyavihar, Mumbai - 77

CERTIFICATE

This is to certify that Jerin J. John, 2nd year student of Computer Engineering Dept., K. J. Somaiya College of Engineering has undergone summer internship on the project Dynamic Query Processing using C#, from 23-05-2016 to 30-06-2016 at Nuclear Information Systems Section, Electronics Division, Bhabha Atomic Research Centre, Trombay under my guidance and supervision.

Smt. Padmini S.Project Guide

ACKNOWLEDGEMENT

I am highly indebted to Smt. Padmini S., SO/H+, NISS, ED, BARC for taking out time from her busy schedule and providing me with continuous guidance, motivation and support throughout the project activity.

I would like to take this opportunity to express my deep gratitude to all staff of NISS, BARC, for their valuable support and suggestions in my project work. Without their advice and help, it would have been very difficult to complete this work.

INDEX

Aim of the Project

Introduction

Language and Concepts

o Delegateso Lambda Expressionso LINQo Dynamic Code Compilation

Project Program Code

Screenshots – Program Output

Aim of the project Develop a program in C# to compile and execute a Linq query at runtime, from a text file or text input, on data stored in any Xml file.

IntroductionThe project focuses on the basic idea of dynamically querying data from a C# program using the concept of LINQ and dynamic code compilation.

A query is a request for information from a database. Depending on the database management system, there are multiple query languages which can be used to perform operations on the data.The most commonly used format is called Structured Query Language [SQL] which provides the platform to access a database and perform operations like insertion, updation, deletion, etc.

C# language supports a technology called LINQ, which stands for Language Integrated Query, which brings all the features and flexibility of SQL right into the programming language. LINQ can be used to query data across multiple platforms from XML to databases. LINQ combined with other C# language concepts like lambdas and delegates, defines a highly efficient way to create and execute queries from programs.

In this project, usage of Linq is demonstrated on data stored in XML files. The data is fetched from the XML file and is stored in the form of a generic List<>, which is then queried using Linq.

The project scope has been expanded to support dynamic querying, i.e., obtain a query either from text file or user input at runtime and operate it on the data.

Dynamic Querying is incorporated by using the Code Compilation assemblies provided by C#, which enables programs to create, compile and execute programs from within themselves. This concept is utilized to create an internal program consisting of a method, which dynamically processes the query on the datalist and returns its resultset.

The execution flow of the program involves:- Reading data and queries from their respective files. Initializing the dynamic code compiler and setting its respective parameters. Defining the internal program and method which is responsible for execution of

the query. Compiling the source code and obtaining an assembly. Creating an instance and calling the internal method to execute the query.

Future ScopeThe scope of this program can be expanded to dynamically query data stored across multiple platforms like databases, DOMs, etc and hence can be applied to several systems.

Language and Concepts

C# is a modern, general-purpose, multi-paradigm programming language encompassing strong typing, imperative,declarative, functional, generic, object-oriented (class-based), and component-oriented programming disciplines developed and maintained by Microsoft and approved by European Computer Manufacturers Association (ECMA) and International Standards Organization (ISO).  C# is designed for Common Language Infrastructure (CLI), which consists of the executable code and runtime environment that allows use of various high-level languages on different computer platforms and architectures.C# constructs closely follow traditional high-level languages, C and C++ being an object-oriented programming language. It has strong resemblance with Java, and has numerous strong programming features that make it endearing to a number of programmers worldwide.

Following is the list of few important features of C#:

Boolean Conditions Automatic Garbage Collection Standard Library Assembly Versioning Properties and Events Delegates and Events Management Easy-to-use Generics Indexers Conditional Compilation Simple Multithreading LINQ and Lambda Expressions Integration with Windows

Delegates

C# delegates are similar to pointers to functions, in C or C++. A delegate is a reference type variable that holds the reference to a method with a particular parameter list and return type. The reference can be changed at runtime. When you instantiate a delegate, you can associate its instance with any method with a compatible signature and return type. You can invoke (or call) the method through the delegate instance.Delegates are used to pass methods as arguments to other methods. Event handlers are nothing more than methods that are invoked through delegates. Any method from any accessible class or struct that matches the delegate type can be assigned to the delegate. The method can be either static or an instance method. This makes it possible to programmatically change method calls, and also plug new code into existing classes.Delegates are especially used for implementing events and the call-back methods. All delegates are implicitly derived from the System.Delegate class.Syntax for delegate declaration is:

delegate<return type><delegate-name><parameter list>

Lambda ExpressionsA lambda expression is an anonymous function that you can use to create delegates or expression tree types. By using lambda expressions, you can write local functions that can be passed as arguments or returned as the value of function calls. Lambda expressions are particularly helpful for writing LINQ query expressions.To create a lambda expression, you specify input parameters (if any) on the left side of the lambda operator =>, and you put the expression or statement block on the other side. For example, the lambda expression x => x * x specifies a parameter that’s named x and returns the value of x squared.Lambdas are used in method-based LINQ queries as arguments to standard query operator methods such as Where<TSource>.

Expression LambdasA lambda expression with an expression on the right side of the => operator is called an expression lambda. Expression lambdas are used extensively in the construction of Expression Trees (C# and Visual Basic). An expression lambda returns the result of the expression and takes the following basic form:

(input parameters) =>expression Eg : (x, y) => x == y

Statement Lambdas

A statement lambda resembles an expression lambda except that the statement(s) is enclosed in braces:

(input parameters) => {statement;}The body of a statement lambda can consist of any number of statements; however, in practice there are typically no more than two or three.

LINQLanguage Integrated Query (LINQ, pronounced "link") is a Microsoft .NET Framework component that adds native data querying capabilities to .NET languages, although ports exist for Java, PHP, JavaScript and ActionScript.LINQ (Language Integrated Query) allows writing queries even without the knowledge of query languages like SQL, XML etc. LINQ queries can be written for diverse data types. Language-integrated query allows query expressions to benefit from the rich metadata, compile-time syntax checking, static typing and IntelliSense that was previously available only to imperative code. Language-integrated query also allows a single general purpose declarative query facility to be applied to all in-memory information, not just information from external sources.

LINQ Architecture in .NETLINQ has a 3-layered architecture in which the uppermost layer consists of the language extensions and the bottom layer consists of data sources that are typically objects implementing IEnumerable<T> or IQueryable<T> generic interfaces. The architecture is shown below in Figure.

Advantages of LINQLINQ offers a host of advantages and among them the foremost is its powerful expressiveness which enables developers to express declaratively. Some of the other advantages of LINQ are given below.

LINQ offers syntax highlighting that proves helpful to find out mistakes during design time.

LINQ offers IntelliSense which means writing more accurate queries easily.

Writing codes is quite faster in LINQ and thus development time also gets reduced significantly.

LINQ makes easy debugging due to its integration in the C# language.

Viewing relationship between two tables is easy with LINQ due to its hierarchical feature and this enables composing queries joining multiple tables in less time.

LINQ allows usage of a single LINQ syntax while querying many diverse data sources and this is mainly because of its unitive foundation.

LINQ is extensible that means it is possible to use knowledge of LINQ to querying new data source types.

LINQ offers the facility of joining several data sources in a single query as well as breaking complex problems into a set of short queries easy to debug.

LINQ offers easy transformation for conversion of one data type to another like transforming SQL data to XML data.

Dynamic Code CompilationDynamic code execution is a powerful feature that allows applications to be extended with code that is not compiled into the application. Users can customize applications and developers can dynamically update code easily. The .Net framework introduces classes that simplifies the tasks by wrapping the details of the process in an easy to use interface that only requires a few lines of code.

.Net provides powerful access to the IL code generation process through the System.CodeDom.Compiler and Microsoft.CSharp and Microsoft.VisualBasic namespaces. In these namespaces you find the tools that allow you to compile an assembly either to disk or into memory. You also need the Reflection namespace as it contains the tools to invoke an object and its methods once you've compiled the object.

The process to execute this code dynamically involves the following steps: Create or read in the code that is to be executed as a string or from file.

Wrap the code into fully functional assembly source code, which includes namespace references (using commands), a namespace and a class that is to be invoked.

Compile the source code into an assembly.

Check for errors on compilation.

Use the assembly reference to create an instance of the object.

Call the specified method on the instance reference returned using Reflection.

Handle any return value from the method call by casting into the proper type.

The CodeDOM (Code Document Object Model) provides types that represent many common types of source code elements. You can design a program that builds a source code model using CodeDOM elements to assemble an object graph. This object graph can be rendered as source code using a CodeDOM code generator for a supported programming language. The CodeDOM can also be used to compile source code into a binary assembly.

Project Code ( C# - program.cs )

using System;using System.Collections;using System.Collections.Generic;using System.IO;using System.Linq;using System.Xml.Linq;using System.Dynamic;using System.CodeDom;using System.CodeDom.Compiler;using System.Linq.Dynamic;using Microsoft.CSharp;

namespace DynQueryProcessor{

class Program {

// .. Initialize dynamic code generation and compilation objects static CSharpCodeProvider cdp = new CSharpCodeProvider(); static CodeCompileUnit ccu = new CodeCompileUnit(); static CompilerParameters cparams = new CompilerParameters(); static CodeEntryPointMethod start = new CodeEntryPointMethod();

static CodeNamespace ns = new CodeNamespace("Internal"); static string filepath = @"F:\Jerin'\Jerin\Codetest5"; static void Main( string[] args ) {

#region Initialisation //Section CodeTypeDeclaration m_class = new CodeTypeDeclaration("Program");

// .. Importing assemblies required for internal code ccu.ReferencedAssemblies.Add("System.dll"); ccu.ReferencedAssemblies.Add("System.Core.dll"); ns.Imports.Add(new CodeNamespaceImport("System")); ns.Imports.Add(new CodeNamespaceImport("System.Collections.Generic")); ns.Imports.Add(new CodeNamespaceImport("System.Linq")); ccu.Namespaces.Add(ns);

Console.ForegroundColor = ConsoleColor.Green;

// .. Load the XML document containing data to be queried XDocument xd = XDocument.Load(@"F:\Jerin'\Jerin\Xmltest.xml"); XElement xe1 = XElement.Load(@"F:\Jerin'\Jerin\Xmltest.xml");

// .. Load the text file containing query to be executed string queryfile = @"F:\Jerin'\Jerin\QueryTest1.txt"; string query = QueryReader(queryfile); //string modq = QueryIsolator(query); Console.Write(" Enter name of field you wish to query (Student/Employee/Department) : "); List<dynamic> datalist = XmlListGen(xd , Console.ReadLine());

start.Comments.Add(new CodeCommentStatement("methodregion")); m_class.Members.Add(start); ns.Types.Add(m_class);

#endregion #region Experiment //Section (LinQ) //var elist = xe1.Elements(); //var resultset = datalist.Where(dept => dept.DName.Contains

("Computer")).Select(dept => dept.Name);

//XmlWriter xw1 = xe1.CreateWriter(); //XmlReader xr1 = xe1.CreateReader(); //xr1.ReadStartElement(); //String s1 = xr1.ReadOuterXml(); //Console.WriteLine(s1); //XmlElement x1 = xd.DocumentElement;

//var slist = elist.Where().Select( file => file );

//var slist1 = elist.Descendants().Where( e => e.Value.Contains( sval ) ).Select( e => e );

//foreach( var item in resultset ) //{ // Console.WriteLine( item ); //} #endregion // .. Add required assemblies to the compiler parameters cparams.ReferencedAssemblies.Add( "System.dll" ); cparams.ReferencedAssemblies.Add( "System.Core.dll" ); cparams.ReferencedAssemblies.Add( "Microsoft.CSharp.dll"); cparams.ReferencedAssemblies.Add( "System.Data.dll"); cparams.ReferencedAssemblies.Add( "System.Xml.dll"); cparams.ReferencedAssemblies.Add( "System.Xml.Linq.dll" ); cparams.GenerateExecutable = false; cparams.GenerateInMemory = true; cparams.TreatWarningsAsErrors = false; cparams.CompilerOptions = "/optimize";

string codefile = CodeFileGenerator(ccu);

// .. Code for internal method to execute the query string methodcode = "\n\t public List<dynamic> QueryFire(List<dynamic> data) \n\t{" + "\n\t\t try {" + "\n\t\t\t var result = " +query+".Cast<dynamic>().ToList();" + "\n\t\t\t return result ;" + "\n\t\t }\n\t\t catch(Exception ex) " + "\n\t\t { \n\t\t\t return new List<dynamic>

{ex.Message + ex.StackTrace}; \n\t\t } \n\t } \n\n";

// .. Replacing comment in internal program with above query method File.WriteAllText(codefile, System.Text.RegularExpressions.Regex.

Replace(File.ReadAllText(codefile), "// methodregion", methodcode));

using (StreamReader sr = new StreamReader(filepath)) { while (!sr.EndOfStream) Console.WriteLine(sr.ReadLine()); } Console.WriteLine("-------------------------------------------------- \n\n\t Query Output :- \n"); #region Execution // Section

// .. Obtain an assembly compiled from the codefile var result = cdp.CompileAssemblyFromFile(cparams, codefile);

// .. Checking internal compiler for any compilation errors if (result.Errors.Count > 0)

{ foreach (CompilerError ce in result.Errors) { Console.WriteLine(" Error : " + ce.ErrorText + " on line " + ce.Line); } } else {

// .. Create instance of internal class and execute the query method dynamic instance = Activator.CreateInstance

(result.CompiledAssembly.GetType("Internal.Program")); dynamic output = instance.QueryFire(datalist); // .. Display the resultset of fired query foreach (var item in output) { Console.WriteLine(item); } } #endregion

Console.Read(); }

// .. Method to read query from text file static string QueryReader( string Filename ) { String query; using( StreamReader sr = new StreamReader( Filename ) ) { query = sr.ReadLine(); } return query; }

// .. Method to return a List<dynamic> from XML data to be queried static List<dynamic> XmlListGen(XDocument xd , string searchfield) { List<dynamic> datalist = new List<dynamic>(); if (searchfield == "Employee") { var elist = xd.Descendants("Employee").Select(s => s); foreach (var item in elist) { dynamic emp = new ExpandoObject(); emp.Ename = item.Element("Ename").Value; emp.Salary = item.Element("Salary").Value; emp.DName = item.Element("DName").Value; emp.EmpID = item.Element("EmpID").Value; datalist.Add(emp); } foreach (var item in elist) {

Console.WriteLine(item.Name + item.Value); } } else if (searchfield == "Student") { var elist = xd.Descendants("Student").Select(s => s); foreach (var item in elist) { dynamic student = new ExpandoObject(); student.Name = item.Element("Name").Value; student.Ssn = item.Element("Ssn").Value; student.DName = item.Element("DName").Value;

datalist.Add(student); } foreach (var item in elist) { Console.WriteLine(item.Name + item.Value); } } else { var elist = xd.Descendants("Department").Select(s => s); foreach (var item in elist) { dynamic dept = new ExpandoObject(); dept.Intake = item.Element("Intake").Value; dept.DName = item.Element("DName").Value; datalist.Add(dept); } foreach (var item in elist) { Console.WriteLine(item.Name + item.Value); }

} return datalist;

} static string QueryIsolator(string query) { Dictionary<string , string> qsegments = new Dictionary<string , string>();

string nminus = query.Substring( query.IndexOf( '.' ) ); string descendantq = null; qsegments.Add( "nminus" , nminus); string listname = query.Remove(query.IndexOf

(qsegments.Where( q => q.Key == "nminus" ).Select( q => q.Value).Single()));

qsegments.Add( "listname" , listname ); if (query.Contains("Descendants"))

{ descendantq = nminus.Substring(nminus.IndexOf("Descendants")); descendantq = descendantq.Remove(descendantq.IndexOf(")")); descendantq = descendantq.Replace("Descendants(", ""); qsegments.Add("desc", descendantq); } if ( query.Contains( "Where" )) { string whereq = nminus.Substring( nminus.IndexOf( "Where" ) ); whereq = whereq.Remove( whereq.IndexOf( ")" ) ); whereq = whereq.Replace( "Where(" , "" ); qsegments.Add( "where" , whereq ); } if( query.Contains( "Select" ) ) { string selectq = nminus.Substring( nminus.IndexOf( "Select" ) ); selectq = selectq.Remove( selectq.IndexOf( ")" ) ); selectq = selectq.Replace( "Select(" , "" ); qsegments.Add( "select" , selectq ); } Console.WriteLine(nminus); return descendantq; }

// .. Method to generate a code file from requested parameters static string CodeFileGenerator(CodeCompileUnit ccu) { CSharpCodeProvider provider = new CSharpCodeProvider(); if (provider.FileExtension[0] != '.') { filepath += "."; } filepath += provider.FileExtension; using (StreamWriter sw = new StreamWriter(filepath, false)) { IndentedTextWriter itw = new IndentedTextWriter(sw); provider.GenerateCodeFromCompileUnit(ccu, itw, new CodeGeneratorOptions()); } return filepath;

} }}

Program Output Screenshots:

Field to be queried.

Dynamically generated internal program containing method - QueryFire()to process the query.

Sample Query outputs.

Graphical User Interface implementation of Project

Select location of XML File and verify using file preview.

Select location of Query File, select domain to operate upon and verify using file preview.

Query Output.

Enter Query into text field by choosing “ Enter Query “ option and select domain to operate upon.

Query Output.