Home » Visual StudioRSS

TFS 2010 Warehouse update - A severe error occurred on the current command - when using custom work

Hi,

We have recently migrated to TFS 2010 and all was going well until a week ago when the tfs_warehouse database stopped getting populated. I know this because our reports are showing old data and the LastUpdatedDateTime is for the FactWorkItemHistory table is showing an old date (13/10/10). The only thing I can think that we did on or around that date is import a custom work item type using witadmin -importwitd. If it would help, I can append the XML definition file of this custom work item to this post.

I have tried the following to try and obtain more information about the problem:-

1) Manually firing the processing jobs using the web service methods: http://tfs2010:8080/tfs/TeamFoundation/Administration/v3.0/WarehouseControlService.asmx

This just returns "true" but the reports still show old data.

2) Checked the event log. This is what I found:

TF53010: The following error has occurred in a Team Foundation component or extension:

Date (UTC): 19/10/2010 10:00:09

Machine: TFS2010

Application Domain: TfsJobAgent.exe

Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727

Service Host:

Process Details:

Process Name: TFSJobAgent

Process Id: 7164

Thread Id: 7780

Account name: <<Domain>>\tfsadmin

Detailed Message: TF221122: An error occurred running job Work Item Tracking Warehouse Sync for team project collection or Team Foundation server <<project>> Exception Message: A severe error occurred on the current command. The results, if any, should be discarded.

Operation cancelled by user. (type SqlExistingConnectionForciblyClosedException)

Exception Stack Trace: at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.TranslateException(SqlException sqlException)

at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.MapException(SqlException ex, QueryExecutionState queryState)

at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.HandleException(SqlException ex)

at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior)

at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(String sqlBatch, List`1 parameterList, Boolean& errorOnBulkUpdate)

at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchQuery(IRequestContext context, String sqlBatch, List`1 parameterList)

at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchInternal(IRequestContext context)

at Microsoft.TeamFoundation.WorkItemTracking.Server.DataAccessLayerImpl.GetWarehouseData(String logicalTrackingId, Int32 batchSize, String fieldLogicalTrackingId, Nullable`1 fieldId, Payload warehousePayload)

at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.TryGetNextWorkItemRevisionsBatch(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, Nullable`1 fieldId, Payload& workItemRevisionBatch)

at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.GetWorkItemRevisionBatchAndPopulate(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, WarehouseMetadata allTpcWarehouseMetadata, String& maxRevisionWatermark)

at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.SynchronizeNewWorkItems(WarehouseMetadata allTpcWarehouseMetadata)

at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.MakeDataChanges()

at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.MakeDataChanges(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String& resultMessage)

at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)

at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)

Inner Exception Details:

Exception Message: A severe error occurred on the current command. The results, if any, should be discarded.

Operation cancelled by user. (type SqlException)

SQL Exception Class: 11

SQL Exception Number: 0

SQL Exception Procedure:

SQL Exception Line Number: 0

SQL Exception Server: TFS2010

SQL Exception State: 0

SQL Error(s):

SQL Error[1]: System.Data.SqlClient.SqlError: Operation cancelled by user.

Class: 11

Number: 0

Server: TFS2010

Source: .Net SqlClient Data Provider

State: 0

Procedure:

Line Number: 0

Exception Data Dictionary:

HelpLink.ProdName = Microsoft SQL Server

HelpLink.ProdVer = 10.50.1600

HelpLink.EvtSrc = MSSQLServer

HelpLink.EvtID = 0

HelpLink.BaseHelpUrl = http://go.microsoft.com/fwlink

HelpLink.LinkId = 20476

Exception Stack Trace: at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)

at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()

at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)

at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()

at System.Data.SqlClient.SqlDataReader.get_MetaData()

at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)

at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)

at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)

at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)

at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)

at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)

at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior)

3) Dropping both the tfs_warehouse and analysis services databases in management studio and recreating them using the TFS 2010 administration console. The recreation worked but I still get the error from above in the event log when the warehouse job runs.

4) Running the GetProcessingStatus to see if any more information was available: http://tfs2010:8080/tfs/TeamFoundation/Administration/v3.0/WarehouseControlService.asmx/GetProcessingStatus

This is the output:

<Job Name="Work Item Tracking Warehouse Sync" JobProcessingStatus="DataChange">

-<LastRun QueueTimeUtc="2010-10-19T09:51:56.3Z" ExecutionStartTimeUtc="2010-10-19T09:51:56.867Z" EndTimeUtc="2010-10-19T10:00:10.31Z" Result="Failed">
 <ResultMessage>[Work Item Tracking Warehouse Sync]: ---> MakeDataChanges() result=DataChangesPending, SchemaChangesPending. ---> MakeSchemaChanges() ---> Microsoft.TeamFoundation.Server.WarehouseException: TF221122: An error occurred running job Work Item Tracking Warehouse Sync for team project collection or Team Foundation server <<SERVER>>. ---> Microsoft.TeamFoundation.WorkItemTracking.Server.SqlExistingConnectionForciblyClosedException: A severe error occurred on the current command. The results, if any, should be discarded. Operation cancelled by user. ---> System.Data.SqlClient.SqlException: A severe error occurred on the current command. The results, if any, should be discarded. Operation cancelled by user. at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning() at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.SqlDataReader.ConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior) --- End of inner exception stack trace --- at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.TranslateException(SqlException sqlException) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.MapException(SqlException ex, QueryExecutionState queryState) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.HandleException(SqlException ex) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior) at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(String sqlBatch, List`1 parameterList, Boolean& errorOnBulkUpdate) at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchQuery(IRequestContext context, String sqlBatch, List`1 parameterList) at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlBatchBuilder.ExecuteBatchInternal(IRequestContext context) at Microsoft.TeamFoundation.WorkItemTracking.Server.DataAccessLayerImpl.GetWarehouseData(String logicalTrackingId, Int32 batchSize, String fieldLogicalTrackingId, Nullable`1 fieldId, Payload warehousePayload) at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.TryGetNextWorkItemRevisionsBatch(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, Nullable`1 fieldId, Payload& workItemRevisionBatch) at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.GetWorkItemRevisionBatchAndPopulate(String workItemRevisionWatermark, String workItemFieldWatermark, Int32 workItemRevisionBatchSize, WarehouseMetadata allTpcWarehouseMetadata, String& maxRevisionWatermark) at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.SynchronizeNewWorkItems(WarehouseMetadata allTpcWarehouseMetadata) at Microsoft.TeamFoundation.WorkItemTracking.Adapter.WorkItemTrackingWarehouseAdapter.MakeDataChanges() at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.MakeDataChanges(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String& resultMessage) at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage) at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage) --- End of inner exception stack trace ---</ResultMessage>
I am now at a loss as to what else to try. I have followed various posts relating to similar issues (such as this one

http://social.msdn.microsoft.com/Forums/en-US/tfsworkitemtracking/thread/991a3e3b-729e-4a49-8e4a-cf3d4a15a685) where they suggested various things such as rebuilding the full text catalog which I have tried with no success. It is only an assumption of mine that the custom work item type is what has broken it but I can't think of anything else that could have caused this issue, bearing in mind that the work item warehousing was working and then it stopped.

 If anyone has any wisdom on this issue, I would appreciate it greatly.

Thanks and kind regards

Daniel Brannan

 

 

1 Answer Found

 

Answer 1

0 down vote

Hello again,

I am delighted to inform everyone that we managed to fix it! The issue was a rogue work item (Bug) which had a link to a Task which did not exist. I am not quite sure how this happened but can only assume it happened during our work item import from TFS 2008.

We only noticed this because, as a last resort, we were going to create a brand new Team Project Collection and Team Project, and import all our work items into it and see if the warehousing worked there. However, when we viewed the 'All Work Items' query as a tree view in Team Explorer prior to the import, one of them was highlighted in red with an exclamation next to it saying the referenced item does not exist. We simply deleted them item using 'witadmin destroywi /collection:http://tfs2010:8080/tfs/<> /id:1571' and then magically the warehousing worked again. Marvellous!

If this post helps even one person then I am a happy man as this has caused us much heartache over the past week. Although we have managed to overcome the issue, it can't be denied that Microsoft's error handling in tfs  leaves a lot to be desired.

Yours

Dan

 
 
 
 
 
 

<< Previous      Next >>

Microsoft   |   Windows   |   Visual Studio   |   Tech Videos   |   Follow us on Twitter