An architecture question

Hi guys,

I got a general architecture question, hope guys can help me. So some basic background first. I have a midsize customer base using my flagship product, which is mainly a crm accounting software. Its a windows based product and they host the data on their servers (usually at their offices).

I have a clickonce application that uploads a subset of their database to our servers. We store all our customer's data in one database. We give each customer and their subsequent records a datasetid.

The clickonce works using a manifest, which essentially figures out which records to add, delete or update on our servers. The transfer of data between the customer's server and our server is through a web service. After the inital upload (which is data intensive), the subsequent uploads only push the difference up (which is much smaller). A majority of our clients tend to upload once day.

We also have a web application that accesses this subset of data to display information to a variety of users, who are our customer's clients. We have about 3000 active users for this web application, which hit the website about once a week.

The issue lie in this, the queries to our database from the website, as well as from the upload service are timing out. Everytime we do a new release, we expand the subset of data (ie add a new table or 2). So immediately after the release we get bombarded with uploads. Hence our site is down alot recently. I think too much traffic is happening to the one database, in terms of data being uploaded and data retrieval for the website.

We would like to keep the subset of data as synchronised with flagship products data as much as possible.

Things we have aleady got include, Scheduler for the clickonce to upload data early in the morning. We are currently implementing a queueing system so only x amount of people can upload at once.

What are some avenues I can explore for the long term and the short term?

Cheer.

From stackoverflow

• One thing I would make sure you look at before you start digging too deep into the optimization process is identifying your bottlenecks. Is it client-side processing, client bandwidth, server bandwidth, or server-side processing. Assuming you're not shipping massive amounts of data around, it's likely server-side processing, but making assumptions like that can get you into trouble on these kinds of things.

• If the uploads don't have to be instantly available (can wait a little bit before you select the data) I'd strongly consider putting a queue in front of that using MSMQ / WCF. That can help you throttle your intake of that data.

  Next it's important to analyze your database structure. There isn't too much information about your schema here, or number of rows, etc... but there are several things I can recommend if you have a lot of incoming data.

  • Make sure that you aren't using a technology that isn't up to the task (like Access or Sqlite).
  • Consider optimizing a database for Inserts, then replicate that database to other databases you might use for selects.
  • If you have a tremendous amount of data, consider setting up a cluster
  • Consider switching to a cloud based data service like Amazon SimpleDB or Azure Data Services

• Where's your bottlneck? Is it a bandwidth issue with your data connection, an issue on your database server, or on your web server, etc?

  What database software are you running? If it's access or something like that it's got to go, in favor or something slightly more enterprise like sql server or similar.

  Are all the clients trying to upload at the same time? Can you stager them somehow, maybe by having them upload at a random time throughout the day, or a randome time during a specific window.

  Would clustering your web server or db server help?

  Can you implement some sort of queue through msmq as already mentioned, or use biztalk or a similar product. That way the client can submit their results, and forget about it, then the queue software handles the actual delivery.

Recaptcha image not showing on IIS

Hello Guys,

I deployed my MVC application in our IIS server. When I go to registration page the recaptcha image is not showing. However, When I run it on Visual Studio its working properly. I'm using IIS 6.

Do you guys have any idea regarding this issue?

Best regards,

From stackoverflow

• Just a guess, but have you set up your public/private keys correctly?

  From the recaptcha API docs

  In order to use reCAPTCHA, you need a public/private API key pair. This key pair helps to prevent an attack where somebody hosts a reCAPTCHA on their website, collects answers from their visitors and submits the answers to your site.

  Your reCAPTCHA token is valid only at the domain you sign up for and any subdomains (due to the potential attack mentioned above).

  As for why it works in development, but not off your server:

  If one of your servers is "localhost" or "127.0.0.1", reCAPTCHA will not enforce the same-domain rule. Just use the same key as for the production server.

  ... so even if you haven't set up your keys properly, the reCAPTCHA will still work if the page is being served from your local machine. Once you deploy though, and incorrect or missing key will prevent the reCAPTCHA from loading.

  samer : Thanks for the reply. Though I haven't tried it yet but definitely your right.

Using UTM with geodjango

I'm looking into using the UTM coordinate system with geodjango. And I can't figure out how to get the data in properly.

I've been browsing the documentation and it seems that the "GEOSGeometry(geo_input, srid=None)" or "OGRGeometry" could be used with an EWKT, but I can't figure out how to format the data.

It looks like the UTM SRID is: 2029

From the wikipedia article the format is written like this:

[UTMZone][N or S] [easting] [northing]

17N 630084 4833438

So I tried the following with no luck:

>>> from django.contrib.gis.geos import *
>>> pnt = GEOSGeometry('SRID=2029;POINT(17N 630084 4833438)')
GEOS_ERROR: ParseException: Expected number but encountered word: '17N'
>>>
>>> from django.contrib.gis.gdal import OGRGeometry
>>> pnt = OGRGeometry('SRID=2029;POINT(17N 630084 4833438)')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python26\lib\site-packages\django\contrib\gis\gdal\geometries.py", line 106, in __init__
    ogr_t = OGRGeomType(geom_input)
  File "C:\Python26\lib\site-packages\django\contrib\gis\gdal\geomtype.py", line 31, in __init__
    raise OGRException('Invalid OGR String Type "%s"' % type_input)
django.contrib.gis.gdal.error.OGRException: Invalid OGR String Type "srid=2029;point(17n 630084 4833438)"

Are there any example available to show how this is done?

May be I should just do any necessary calulations in UTM and convert to decimal degrees?
In this case does GEOS or other tools in geodjango provide convertion utitilites?

From stackoverflow

• The UTM zone (17N) is already specified by the spatial reference system -- SRID 2029, so you don't need to include it in the WKT you pass to the GEOSGeometry constructor.

  >>> from django.contrib.gis.geos import *
  >>> pnt = GEOSGeometry('SRID=2029;POINT(630084 4833438)')
  >>> (pnt.x, pnt.y)
  (630084.0, 4833438.0)
  >>> pnt.srid
  2029
  

  Then, for example:

  >>> pnt.transform(4326)   # Transform to WGS84
  >>> (pnt.x, pnt.y)
  (-79.387137066054038, 43.644504290860461)
  >>> pnt.srid
  4326
  

  mloskot : I would add that geometry given in form of Well-Known-Text, either OGC WKT or PostGIS/GEOS-specific format of Extended WKT, components given between parenthesis () after geometry tag, should only specify coordinates: X Y, optionally Z and M dimensions. No other exotic elements are allowed, so 17N is an invalid token. In case of EWKT, spatial reference system can be given using dedicated specifier SRID.

What is the best way to handle returning multiple formats in ASP.NET MVC

Rails has a nice idiom that makes it easy for you to have a single action method return properly formated data (json, xml, just the data) based on the format specified by the client (or else deduced from the request. It looks something like this ...

respond_to do |format|
  format.html #edit.html.erb
  format.json {render :text=> <your json here>), :layout=> false}
  format.xml ...

end

What is the preferred way to do this in ASP.NET MVC? Ideally, I'd like the framework to work in the same way as Rails (e.g. be able to return the ViewData properly formatted for the format specified by the client or else deduced from the request itself).

Rails also allows you to create views that are specific to each type giving you the opportunity to essentially return the same data to all views and letting them handle formatting the data correctly (so you have a view that builds xml, another that builds json and yet another that builds html). Is this possible with ASP.NET MVC? In fact, this model seems to best keep with the goal of separating concerns imho as it lets the controllers return view-agnostic data whereas most approaches I see today (including the above line "format.json .... :layout => false") do the JSON conversion inside the controller and return that data directly to the client given a request for that format.

Anyhow ... suggestions, thoughts, recommendations?

Thanks

From stackoverflow

• In ASP.NET MVC, controller actions generally return objects that derive from ActionResult, which is then invoked by the runtime while generating the response stream.

  Out-of-the-box you have several classes that derive from ActionResult - ContentResult for text results, ViewResult for content from a view, JsonResult for serializing an object hierarchy into JSON, RedirectResult for redirecting, and so on.

  Generally you pass in the model to the result and let it decide how to generate the result, but it doesn't have to be the same model - I can pass a different object to each result if necessary.

  The concrete type of result returned by an action is not 'baked in' to the action's signature - you can easily pass in a format parameter to your action, and have it generate and return a different ActionResult accordingly:

  public ActionResult ListProducts(string format)
  {
      List<Product> products = ProductService.GetAllProducts();
      if (format == "JSON")
      {
          // eg., transform model for JSON consumption
          List<JsonProduct> jsonProducts = ProductService.ToJSON(products); 
          return Json(jsonProducts);
      }
      else if (format == "XML")
      {
          return new XmlResult(products);
      }
  
     // default is to return HTML from view, which expects List<Product> for model
     return View(products);
  
  
  }

  Note that the methods Json() and View() are built into the controller and are convenience methods for returning JsonResult and ViewResult respectively. XmlResult is an example of a custom ActionResult that takes an object, serializes it to XML, then returns the result as an XML stream.

  The example is a little contrived, but it shows that the controller orchestrates all the work of selecting the result and constructing / transforming the model that is passed to that result. Controller actions should still be light-weight though so you offload the heavy tasks to services, such as loading the model from the business layer, or transforming objects in one model to objects in another model, such as for JSON consumption.

  wgpubs : The only problem here is that rarely is it so easy to just return Json (your model). Typically, I'd expect folks to be using something like JQuery or ExtJs in the client that requires the Json (or xml) to be structured in a particular way. So ... where does that restructuring happen? If we should do it in the controller ... where/how? Also, if we do it there ... does this not violate "separation of concerns"? If we do it in the View ... how do we modify the framework to intelligibly pick the right view based on the requested format? Thanks.

  Sam : I agree, most of the time you will need a separate model for working with JSON. I don't see why separation of concerns is violated though - the controller's concern is to provide the model to the view, which includes working out what that model should be, and which view should be given that model. It's entirely relevant for the controller to do that. It would definitely be a violation if the view was doing the work of the controller. I edited my sample to indicate model construction.

  wgpubs : I guess I'm kind of torn on this one because yes, the controller should be responsible for how the data is packaged and sent of to the view ... but does that mean it should be responsible for such if that means returning the data in potentially many formats based on one or many client-side frameworks being used (e.g. ExtJs, Jquery, Mootools, etc...)? You see, we aren't just returning a bunch of data here ... we are returning data FOR JQuery or FOR ExtJs. That is where the separation thing gets a bit more obscured and also unit testing controllers a bit more hairy.

  Sam : Why would you need to return different formats for different JavaScript frameworks? Surely they can all consume the same JSON? It's just objects, after all - can you give an example of where this would be necessary?

  wgpubs : Nope. The JQuery and ExtJs widgets all have detailed specifications for what the JSON needs to look like ... and they aren't uniform and thus, my question: "By returning JSON in a format suitable for a given framework are we not violating proper separation of concerns?" Now I know most folks are following your approach here as it is the most straightforward ... but I'm wondering if it might be best to return the *same* DTO regardless of request type and define diff. views per request type (e.g. index_json.ascx, index_xml.ascx, index.aspx)? Seems more testable and prop. separated to me.

creating/editing textfile through vb.net

how do i code in vb.net the algorithm below. tnx!

Procedure logfile()
{
    if "C:\textfile.txt"=exist then
        open the textfile;
    else
        create the textfile;
    end if  
    go to the end of the textfile;
    write new line in the textfile;
    save;
    close;
}

From stackoverflow

• Dim FILE_NAME As String = "C:\textfile.txt"
  Dim i As Integer
  Dim aryText(4) As String
  
  aryText(0) = "Mary WriteLine"
  aryText(1) = "Had"
  aryText(2) = "Another"
  aryText(3) = "Little"
  aryText(4) = "One"
  
  Dim objWriter As New System.IO.StreamWriter(FILE_NAME, True)
  
  For i = 0 To 4
      objWriter.WriteLine(aryText(i))
  Next
  
  objWriter.Close()
  MsgBox("Text Appended to the File")
  

  If you set the second parameter to True in the System.IO.StreamWriter's constructor it will append to a file if it already exists, or create a new one if it doesn't.

• Giving you the code will be very easy. But here is a nice article which has code as well as some good explanation. : Reading and writing text files with VB.NET

  Cyril Gupta : +1 I agree, give the guy a fishing reel instead of a fish.

• It's best to use a component that does this type of logging out of the box. The Logging Application Block from Enterprise Library for example. That way, you get flexibility, scalability and don't have contention with your log file.

  To answer your question specifically (sorry, I don't know VB, but the translation should be simple enough) ...

  void Main()
  {
      using( var fs = File.Open( @"c:\textfile.txt", FileMode.Append ) )
      {
       using( var sw = new StreamWriter( fs ) )
       {
         sw.WriteLine( "New Line" );
         sw.Close();
       }
  
       fs.Close();
      }
  }
  

• This can be achieved in a single line too:

  System.IO.File.AppendAllText(filePath, "Hello World" & vbCrLf)
  

  It will create the file if missing, append the text and close it again.

  See MSDN, File.AppendAllText Method.

mysql LIKE returning only one row when three match.

When I query my database with the following in my file, search.php, it only returns the first result it comes across.

$qry = "SELECT business_id FROM business WHERE zip like '%91326%'";

$rs = mysql_query($qry);
$rec = mysql_fetch_array($rs);
echo $session->showContents($rec);

showContents is just a utility function...

function showContents($array)
{
        echo "<pre>";
        print_r($array);
        echo "</pre>";
}

showContents returned this:

Array
(
    [0] => 3
    [business_id] => 3
)

The crazy thing is, when I put the same query in sqlbuddy it gives me:

business_id
3
5
6

I am at a loss

From stackoverflow

• mysql_fetch_array fetches only a single row. You want to use it several times to build an array with the entire result set:

  $rec = array();
  
  while(($row = mysql_fetch_array($rs)) !== FALSE) {
      $rec[] = $row;
  }
  

  If you just want the ID's you want to select the ID:

  $rec = array();
  
  while(($row = mysql_fetch_array($rs)) !== FALSE) {
      $rec[] = $row[0];
  }
  

• Try this:

  $qry = "SELECT business_id FROM business WHERE zip like '%91326%'";
  $rs = mysql_query($qry);
  while ($rec = mysql_fetch_array($rs)) {
      echo $session->showContents($rec);
  }
  

• That's because mysql_fetch_array only fetches a single row from the result set.

  Typically you use it like this (from the manual):

  while ($row = mysql_fetch_array($result, MYSQL_NUM)) {
      printf("ID: %s  Name: %s", $row[0], $row[1]);  
  }
  

jQuery post parameter to load

How do I pass the values of txtname and tel as variables to the .load???

$(document).ready(function(){
    $("#add").click(function(){
        $("#result").load("add.php", {name: #txtname});
    });
});

The html:

<p>Name:<input type="text" name="name" value="" id="txtname" /></p>
<p>Telephone:<input type="text" name="tel" id="tel" value="" /></p>
<input type="submit" value="Submit" id="add" />

From stackoverflow

• $(document).ready(function(){
    $("#add").click(function(){
      $("#result").load("add.php", {
        'name': $("#txtname").val(), 
        'telephone': $("#tel").val()
      });
    });
  });
  

• $(document).ready(function() {
      $("#add").click(function() {
         $("#result").load("add.php", {
             name: $("#txtname").val(), 
             tel: $("#tel").val()
         });
      });
  });