Failed jobs do not become expired to allow you to re-queue them without any
time pressure. You should re-queue or delete them manually, or apply AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete)
attribute to delete them automatically.
Failed to change state to a 'Succeeded' one due to an exception after 10 retry attempts Fewer details...
05/08/2026 18:11:26
System.Data.SqlClient.SqlException
Could not allocate space for object 'HangFire.Job'.'IX_HangFire_Job_ExpireAt' in database 'uat_paydirect_ent' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
System.Data.SqlClient.SqlException (0x80131904): Could not allocate space for object 'HangFire.Job'.'IX_HangFire_Job_ExpireAt' in database 'uat_paydirect_ent' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
at System.Data.SqlClient.SqlConnection.OnError(SqlExceptionexception, BooleanbreakConnection, Action`1wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObjectstateObj, BooleancallerHasConnectionLock, BooleanasyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehaviorrunBehavior, SqlCommandcmdHandler, SqlDataReaderdataStream, BulkCopySimpleResultSetbulkCopyHandler, TdsParserStateObjectstateObj, Boolean&dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReaderds, RunBehaviorrunBehavior, StringresetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehaviorcmdBehavior, RunBehaviorrunBehavior, BooleanreturnStream, Booleanasync, Int32timeout, Task&task, BooleanasyncWrite, SqlDataReaderds)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1completion, BooleansendToPipe, Int32timeout, BooleanasyncWrite, StringmethodName)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Hangfire.SqlServer.SqlCommandBatch.ExecuteNonQuery()
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<Commit>b__12_0(DbConnectionconnection, DbTransactiontransaction)
at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass34_0.<UseTransaction>b__0(DbConnectionconnection, DbTransactiontransaction)
at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass35_0`1.<UseTransaction>b__0(DbConnectionconnection)
at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnectiondedicatedConnection, Func`2func)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction(DbConnectiondedicatedConnection, Action`2action)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit()
at Hangfire.States.BackgroundJobStateChanger.ChangeState(StateChangeContextcontext)
at Hangfire.Server.Worker.TryChangeState(BackgroundProcessContextcontext, IStorageConnectionconnection, IFetchedJobfetchedJob, IStatestate, String[]expectedStates, CancellationTokeninitializeToken, CancellationTokenabortToken)
Failed to change state to a 'Succeeded' one due to an exception after 10 retry attempts More details...
05/08/2026 18:11:26
System.Data.SqlClient.SqlException
Could not allocate space for object 'HangFire.Job'.'IX_HangFire_Job_ExpireAt' in database 'uat_paydirect_ent' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
System.Data.SqlClient.SqlException (0x80131904): Could not allocate space for object 'HangFire.Job'.'IX_HangFire_Job_ExpireAt' in database 'uat_paydirect_ent' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
at System.Data.SqlClient.SqlConnection.OnError(SqlExceptionexception, BooleanbreakConnection, Action`1wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObjectstateObj, BooleancallerHasConnectionLock, BooleanasyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehaviorrunBehavior, SqlCommandcmdHandler, SqlDataReaderdataStream, BulkCopySimpleResultSetbulkCopyHandler, TdsParserStateObjectstateObj, Boolean&dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReaderds, RunBehaviorrunBehavior, StringresetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehaviorcmdBehavior, RunBehaviorrunBehavior, BooleanreturnStream, Booleanasync, Int32timeout, Task&task, BooleanasyncWrite, SqlDataReaderds)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1completion, BooleansendToPipe, Int32timeout, BooleanasyncWrite, StringmethodName)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Hangfire.SqlServer.SqlCommandBatch.ExecuteNonQuery()
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<Commit>b__12_0(DbConnectionconnection, DbTransactiontransaction)
at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass34_0.<UseTransaction>b__0(DbConnectionconnection, DbTransactiontransaction)
at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass35_0`1.<UseTransaction>b__0(DbConnectionconnection)
at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnectiondedicatedConnection, Func`2func)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction(DbConnectiondedicatedConnection, Action`2action)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit()
at Hangfire.States.BackgroundJobStateChanger.ChangeState(StateChangeContextcontext)
at Hangfire.Server.Worker.TryChangeState(BackgroundProcessContextcontext, IStorageConnectionconnection, IFetchedJobfetchedJob, IStatestate, String[]expectedStates, CancellationTokeninitializeToken, CancellationTokenabortToken)
Failed to change state to a 'Succeeded' one due to an exception after 10 retry attempts More details...
05/08/2026 18:11:55
System.Data.SqlClient.SqlException
Could not allocate space for object 'HangFire.Job'.'IX_HangFire_Job_ExpireAt' in database 'uat_paydirect_ent' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
System.Data.SqlClient.SqlException (0x80131904): Could not allocate space for object 'HangFire.Job'.'IX_HangFire_Job_ExpireAt' in database 'uat_paydirect_ent' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
at System.Data.SqlClient.SqlConnection.OnError(SqlExceptionexception, BooleanbreakConnection, Action`1wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObjectstateObj, BooleancallerHasConnectionLock, BooleanasyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehaviorrunBehavior, SqlCommandcmdHandler, SqlDataReaderdataStream, BulkCopySimpleResultSetbulkCopyHandler, TdsParserStateObjectstateObj, Boolean&dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReaderds, RunBehaviorrunBehavior, StringresetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehaviorcmdBehavior, RunBehaviorrunBehavior, BooleanreturnStream, Booleanasync, Int32timeout, Task&task, BooleanasyncWrite, SqlDataReaderds)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1completion, BooleansendToPipe, Int32timeout, BooleanasyncWrite, StringmethodName)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Hangfire.SqlServer.SqlCommandBatch.ExecuteNonQuery()
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<Commit>b__12_0(DbConnectionconnection, DbTransactiontransaction)
at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass34_0.<UseTransaction>b__0(DbConnectionconnection, DbTransactiontransaction)
at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass35_0`1.<UseTransaction>b__0(DbConnectionconnection)
at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnectiondedicatedConnection, Func`2func)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction(DbConnectiondedicatedConnection, Action`2action)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit()
at Hangfire.States.BackgroundJobStateChanger.ChangeState(StateChangeContextcontext)
at Hangfire.Server.Worker.TryChangeState(BackgroundProcessContextcontext, IStorageConnectionconnection, IFetchedJobfetchedJob, IStatestate, String[]expectedStates, CancellationTokeninitializeToken, CancellationTokenabortToken)
An exception occurred during performance of the job. More details...
04/16/2026 17:05:04
System.InvalidOperationException
Could not get a value of the job parameter `CurrentCulture`. See inner exception for details.
System.InvalidOperationException: Could not get a value of the job parameter `CurrentCulture`. See inner exception for details.
---> System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnectionowningConnection, TaskCompletionSource`1retry, DbConnectionOptionsuserOptions, DbConnectionInternaloldConnection, DbConnectionInternal&connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnectionouterConnection, DbConnectionFactoryconnectionFactory, TaskCompletionSource`1retry, DbConnectionOptionsuserOptions)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1retry)
at System.Data.SqlClient.SqlConnection.Open()
at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()
at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnectiondedicatedConnection, Func`2func)
at Hangfire.Server.PerformContext.GetJobParameter[T](Stringname)
--- End of inner exception stack trace ---
at Hangfire.Server.PerformContext.GetJobParameter[T](Stringname)
at Hangfire.CaptureCultureAttribute.OnPerforming(PerformingContextfilterContext)
at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1tuple)
at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstanceinstance, Func`2action, Stringmessage)
at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfilerprofiler, TInstanceinstance, Action`1action, Stringmessage)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilterfilter, PerformingContextpreContext, Func`1continuation)
An exception occurred during processing of a background job. More details...
04/16/2026 17:05:13
System.InvalidOperationException
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnectionowningConnection, TaskCompletionSource`1retry, DbConnectionOptionsuserOptions, DbConnectionInternaloldConnection, DbConnectionInternal&connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnectionouterConnection, DbConnectionFactoryconnectionFactory, TaskCompletionSource`1retry, DbConnectionOptionsuserOptions)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1retry)
at System.Data.SqlClient.SqlConnection.Open()
at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()
at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnectiondedicatedConnection, Func`2func)
at Hangfire.Server.Worker.PerformJob(BackgroundProcessContextcontext, IStorageConnectionconnection, StringjobId)
An exception occurred during processing of a background job. More details...
04/10/2026 00:16:51
System.Data.SqlClient.SqlException
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 35 - An internal exception was caught)
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 35 - An internal exception was caught)
---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.
at System.Net.Sockets.Socket.ThrowObjectDisposedException()
at System.Net.Sockets.Socket.SetSocketOption(SocketOptionLeveloptionLevel, SocketOptionNameoptionName, Int32optionValue)
at System.Data.SqlClient.SNI.SNITCPHandle..ctor(StringserverName, Int32port, Int64timerExpire, ObjectcallbackObject, Booleanparallel)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnectionowningObject, UInt32waitForMultipleObjectsTimeout, BooleanallowCreate, BooleanonlyOneCheckConnection, DbConnectionOptionsuserOptions, DbConnectionInternal&connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnectionowningObject, TaskCompletionSource`1retry, DbConnectionOptionsuserOptions, DbConnectionInternal&connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnectionowningConnection, TaskCompletionSource`1retry, DbConnectionOptionsuserOptions, DbConnectionInternaloldConnection, DbConnectionInternal&connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnectionouterConnection, DbConnectionFactoryconnectionFactory, TaskCompletionSource`1retry, DbConnectionOptionsuserOptions)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1retry)
at System.Data.SqlClient.SqlConnection.Open()
at Dapper.SqlMapper.QueryImpl[T](IDbConnectioncnn, CommandDefinitioncommand, TypeeffectiveType)+MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1source)
at Hangfire.SqlServer.SqlServerConnection.<>c__DisplayClass8_0.<GetJobData>b__0(DbConnectionconnection)
at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnectiondedicatedConnection, Func`2func)
at Hangfire.Server.Worker.PerformJob(BackgroundProcessContextcontext, IStorageConnectionconnection, StringjobId)
An exception occurred during performance of the job. More details...
02/24/2026 22:45:12
System.InvalidOperationException
This SqlTransaction has completed; it is no longer usable.
System.InvalidOperationException: This SqlTransaction has completed; it is no longer usable.
at System.Data.SqlClient.SqlTransaction.ZombieCheck()
at System.Data.SqlClient.SqlTransaction.Rollback()
at TqsNibssPDService.BackgroundOperations.Repositories.EntPaymentRepository.LogSoftAlliancePendingItems(List`1items, EntPaymentSoftAlliancepayment, StringprimaryItemCode, DecimalbillAmount) in /app/TqsNibssPDService/BackgroundOperations/Repositories/EntPaymentRepository.cs:line 379
at TqsNibssPDService.BackgroundOperations.Operations.PaymentItemsLoggerProcessor.ExecuteForSoftAlliance(EntPaymentSoftAlliancepayment) in /app/TqsNibssPDService/BackgroundOperations/Operations/PaymentItemsLoggerProcessor.cs:line 61
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
02/24/2026 22:30:09
System.InvalidOperationException
This SqlTransaction has completed; it is no longer usable.
System.InvalidOperationException: This SqlTransaction has completed; it is no longer usable.
at System.Data.SqlClient.SqlTransaction.ZombieCheck()
at System.Data.SqlClient.SqlTransaction.Rollback()
at TqsNibssPDService.BackgroundOperations.Repositories.EntPaymentRepository.LogSoftAlliancePendingItems(List`1items, EntPaymentSoftAlliancepayment, StringprimaryItemCode, DecimalbillAmount) in /app/TqsNibssPDService/BackgroundOperations/Repositories/EntPaymentRepository.cs:line 379
at TqsNibssPDService.BackgroundOperations.Operations.PaymentItemsLoggerProcessor.ExecuteForSoftAlliance(EntPaymentSoftAlliancepayment) in /app/TqsNibssPDService/BackgroundOperations/Operations/PaymentItemsLoggerProcessor.cs:line 61
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
02/24/2026 22:15:18
System.InvalidOperationException
This SqlTransaction has completed; it is no longer usable.
System.InvalidOperationException: This SqlTransaction has completed; it is no longer usable.
at System.Data.SqlClient.SqlTransaction.ZombieCheck()
at System.Data.SqlClient.SqlTransaction.Rollback()
at TqsNibssPDService.BackgroundOperations.Repositories.EntPaymentRepository.LogSoftAlliancePendingItems(List`1items, EntPaymentSoftAlliancepayment, StringprimaryItemCode, DecimalbillAmount) in /app/TqsNibssPDService/BackgroundOperations/Repositories/EntPaymentRepository.cs:line 379
at TqsNibssPDService.BackgroundOperations.Operations.PaymentItemsLoggerProcessor.ExecuteForSoftAlliance(EntPaymentSoftAlliancepayment) in /app/TqsNibssPDService/BackgroundOperations/Operations/PaymentItemsLoggerProcessor.cs:line 61
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
02/24/2026 22:00:28
System.InvalidOperationException
This SqlTransaction has completed; it is no longer usable.
System.InvalidOperationException: This SqlTransaction has completed; it is no longer usable.
at System.Data.SqlClient.SqlTransaction.ZombieCheck()
at System.Data.SqlClient.SqlTransaction.Rollback()
at TqsNibssPDService.BackgroundOperations.Repositories.EntPaymentRepository.LogSoftAlliancePendingItems(List`1items, EntPaymentSoftAlliancepayment, StringprimaryItemCode, DecimalbillAmount) in /app/TqsNibssPDService/BackgroundOperations/Repositories/EntPaymentRepository.cs:line 379
at TqsNibssPDService.BackgroundOperations.Operations.PaymentItemsLoggerProcessor.ExecuteForSoftAlliance(EntPaymentSoftAlliancepayment) in /app/TqsNibssPDService/BackgroundOperations/Operations/PaymentItemsLoggerProcessor.cs:line 61
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)