Generic stochastic gradient descent training for kernel-based models. More...

#include <shark/Algorithms/Trainers/KernelSGDTrainer.h>

Inheritance diagram for shark::KernelSGDTrainer< InputType, CacheType >:

Public Types
typedef AbstractKernelFunction< InputType >	KernelType

typedef KernelClassifier< InputType >	ClassifierType

typedef KernelExpansion< InputType >	ModelType

typedef AbstractLoss< unsigned int, RealVector >	LossType

typedef ConstProxyReference< typenameBatch< InputType >::typeconst >::type	ConstBatchInputReference

typedef CacheType	QpFloatType

typedef KernelMatrix< InputType, QpFloatType >	KernelMatrixType

typedef PartlyPrecomputedMatrix< KernelMatrixType >	PartlyPrecomputedMatrixType

Public Types inherited from shark::AbstractTrainer< KernelClassifier< InputType > >
typedef KernelClassifier< InputType >	ModelType

typedef ModelType::InputType	InputType

typedef typename Model::OutputType	LabelType

typedef LabeledData< InputType, LabelType >	DatasetType

Public Types inherited from shark::IParameterizable< VectorType >
typedef VectorType	ParameterVectorType

Public Member Functions
	KernelSGDTrainer (KernelType kernel, const LossType loss, double C, bool offset, bool unconstrained=false, size_t cacheSize=0x4000000)
	Constructor.

double	cacheSize () const
	return current cachesize

void	setCacheSize (std::size_t size)

std::string	name () const
	From INameable: return the class name.

void	train (ClassifierType &classifier, const LabeledData< InputType, unsigned int > &dataset)

std::size_t	epochs () const

void	setEpochs (std::size_t value)

KernelType *	kernel ()
	get the kernel function

const KernelType *	kernel () const
	get the kernel function

void	setKernel (KernelType *kernel)
	set the kernel function

bool	isUnconstrained () const

double	C () const
	return the value of the regularization parameter

void	setC (double value)
	set the value of the regularization parameter (must be positive)

bool	trainOffset () const
	check whether the model to be trained should include an offset term

RealVector	parameterVector () const
	Returns the vector of hyper-parameters.

void	setParameterVector (RealVector const &newParameters)
	Sets the vector of hyper-parameters.

size_t	numberOfParameters () const
	Returns the number of hyper-parameters.

Public Member Functions inherited from shark::AbstractTrainer< KernelClassifier< InputType > >
virtual void	train (ModelType &model, DatasetType const &dataset)=0
	Core of the Trainer interface.

Public Member Functions inherited from shark::INameable
virtual	~INameable ()

Public Member Functions inherited from shark::ISerializable
virtual	~ISerializable ()
	Virtual d'tor.

virtual void	read (InArchive &archive)
	Read the component from the supplied archive.

virtual void	write (OutArchive &archive) const
	Write the component to the supplied archive.

void	load (InArchive &archive, unsigned int version)
	Versioned loading of components, calls read(...).

void	save (OutArchive &archive, unsigned int version) const
	Versioned storing of components, calls write(...).

	BOOST_SERIALIZATION_SPLIT_MEMBER ()

Public Member Functions inherited from shark::IParameterizable< VectorType >
virtual	~IParameterizable ()

Protected Attributes
KernelType *	m_kernel
	pointer to kernel function

const LossType *	m_loss
	pointer to loss function

double	m_C
	regularization parameter

bool	m_offset
	should the resulting model have an offset term?

bool	m_unconstrained
	should C be stored as log(C) as a parameter?

std::size_t	m_epochs
	number of training epochs (sweeps over the data), or 0 for default = max(10, C)

std::size_t	m_cacheSize

Detailed Description

template<class InputType, class CacheType = float>
class shark::KernelSGDTrainer< InputType, CacheType >

Generic stochastic gradient descent training for kernel-based models.

Given a differentiable loss function L(f, y) for classification this trainer solves the regularized risk minimization problem

\[ \min \frac{1}{2} \sum_j \|w_j\|^2 + C \sum_i L(y_i, f(x_i)), \]

where i runs over training data, j over classes, and C > 0 is the regularization parameter.

This implementation is an adaptation of the PEGASOS algorithm, see the paper Shalev-Shwartz et al. "Pegasos: Primal estimated sub-gradient solver for SVM." Mathematical Programming 127.1 (2011): 3-30.

However, the (non-essential) projection step is dropped, and the algorithm is applied to a kernelized model. The resulting optimization scheme amounts to plain standard stochastic gradient descent (SGD) with update steps of the form

\[ w_j \leftarrow (1 - 1/t) w_j + \frac{C}{t} \frac{\partial L(y_i, f(x_i))}{\partial w_j} \]

for random index i. The only notable trick borrowed from that paper is the representation of the weight vectors in the form

\[ w_j = s \cdot \sum_i \alpha_{i,j} k(x_i, \cdot) \]

with a scalar factor s (called alphaScale in the code). This enables scaling with factor (1 - 1/t) in constant time.

: NOTE: Being an SGD-based solver, this algorithm is relatively fast for differentiable loss functions such as the logistic loss (class CrossEntropy). It suffers from significantly slower convergence for non-differentiable losses, e.g., the hinge loss for SVM training.

Definition at line 90 of file KernelSGDTrainer.h.

Member Typedef Documentation

◆ ClassifierType

template<class InputType , class CacheType = float>

typedef KernelClassifier<InputType> shark::KernelSGDTrainer< InputType, CacheType >::ClassifierType

Definition at line 95 of file KernelSGDTrainer.h.

◆ ConstBatchInputReference

template<class InputType , class CacheType = float>

typedef ConstProxyReference<typenameBatch<InputType>::typeconst>::type shark::KernelSGDTrainer< InputType, CacheType >::ConstBatchInputReference

Definition at line 98 of file KernelSGDTrainer.h.

◆ KernelMatrixType

template<class InputType , class CacheType = float>

typedef KernelMatrix<InputType, QpFloatType> shark::KernelSGDTrainer< InputType, CacheType >::KernelMatrixType

Definition at line 101 of file KernelSGDTrainer.h.

◆ KernelType

template<class InputType , class CacheType = float>

typedef AbstractKernelFunction<InputType> shark::KernelSGDTrainer< InputType, CacheType >::KernelType

Definition at line 94 of file KernelSGDTrainer.h.

◆ LossType

template<class InputType , class CacheType = float>

typedef AbstractLoss<unsigned int, RealVector> shark::KernelSGDTrainer< InputType, CacheType >::LossType

Definition at line 97 of file KernelSGDTrainer.h.

◆ ModelType

template<class InputType , class CacheType = float>

typedef KernelExpansion<InputType> shark::KernelSGDTrainer< InputType, CacheType >::ModelType

Definition at line 96 of file KernelSGDTrainer.h.

◆ PartlyPrecomputedMatrixType

template<class InputType , class CacheType = float>

typedef PartlyPrecomputedMatrix< KernelMatrixType > shark::KernelSGDTrainer< InputType, CacheType >::PartlyPrecomputedMatrixType

Definition at line 102 of file KernelSGDTrainer.h.

◆ QpFloatType

template<class InputType , class CacheType = float>

typedef CacheType shark::KernelSGDTrainer< InputType, CacheType >::QpFloatType

Definition at line 99 of file KernelSGDTrainer.h.

Constructor & Destructor Documentation

◆ KernelSGDTrainer()

template<class InputType , class CacheType = float>

shark::KernelSGDTrainer< InputType, CacheType >::KernelSGDTrainer	(	KernelType *	kernel,
		const LossType *	loss,
		double	C,
		bool	offset,
		bool	unconstrained = `false`,
		size_t	cacheSize = `0x4000000`
	)

inline

Constructor.

Parameters

kernel	kernel function to use for training and prediction
loss	(sub-)differentiable loss function
C	regularization parameter - always the 'true' value of C, even when unconstrained is set
offset	whether to train with offset/bias parameter or not
unconstrained	when a C-value is given via setParameter, should it be piped through the exp-function before using it in the solver?
cacheSize	size of the cache

Definition at line 113 of file KernelSGDTrainer.h.

Member Function Documentation

◆ C()

template<class InputType , class CacheType = float>

double shark::KernelSGDTrainer< InputType, CacheType >::C ( ) const

inline

return the value of the regularization parameter

Definition at line 224 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_C.

◆ cacheSize()

template<class InputType , class CacheType = float>

double shark::KernelSGDTrainer< InputType, CacheType >::cacheSize ( ) const

inline

return current cachesize

Definition at line 125 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_cacheSize.

◆ epochs()

template<class InputType , class CacheType = float>

std::size_t shark::KernelSGDTrainer< InputType, CacheType >::epochs ( ) const

inline

Return the number of training epochs. A value of 0 indicates that the default of max(10, C) should be used.

Definition at line 199 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_epochs.

◆ isUnconstrained()

template<class InputType , class CacheType = float>

bool shark::KernelSGDTrainer< InputType, CacheType >::isUnconstrained ( ) const

inline

check whether the parameter C is represented as log(C), thus, in a form suitable for unconstrained optimization, in the parameter vector

Definition at line 220 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_unconstrained.

◆ kernel() [1/2]

template<class InputType , class CacheType = float>

KernelType * shark::KernelSGDTrainer< InputType, CacheType >::kernel ( )

inline

get the kernel function

Definition at line 208 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_kernel.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::setKernel().

◆ kernel() [2/2]

template<class InputType , class CacheType = float>

const KernelType * shark::KernelSGDTrainer< InputType, CacheType >::kernel ( ) const

inline

get the kernel function

Definition at line 211 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_kernel.

◆ name()

template<class InputType , class CacheType = float>

std::string shark::KernelSGDTrainer< InputType, CacheType >::name ( ) const

inlinevirtual

From INameable: return the class name.

Reimplemented from shark::INameable.

Definition at line 137 of file KernelSGDTrainer.h.

◆ numberOfParameters()

template<class InputType , class CacheType = float>

size_t shark::KernelSGDTrainer< InputType, CacheType >::numberOfParameters ( ) const

inlinevirtual

Returns the number of hyper-parameters.

Reimplemented from shark::IParameterizable< VectorType >.

Definition at line 254 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_kernel, and shark::IParameterizable< VectorType >::numberOfParameters().

◆ parameterVector()

template<class InputType , class CacheType = float>

RealVector shark::KernelSGDTrainer< InputType, CacheType >::parameterVector ( ) const

inlinevirtual

Returns the vector of hyper-parameters.

Reimplemented from shark::IParameterizable< VectorType >.

Definition at line 239 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_C, shark::KernelSGDTrainer< InputType, CacheType >::m_kernel, shark::KernelSGDTrainer< InputType, CacheType >::m_unconstrained, and shark::IParameterizable< VectorType >::parameterVector().

◆ setC()

template<class InputType , class CacheType = float>

void shark::KernelSGDTrainer< InputType, CacheType >::setC ( double value )

inline

set the value of the regularization parameter (must be positive)

Definition at line 228 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_C, and RANGE_CHECK.

◆ setCacheSize()

template<class InputType , class CacheType = float>

void shark::KernelSGDTrainer< InputType, CacheType >::setCacheSize ( std::size_t size )

inline

Definition at line 131 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_cacheSize.

◆ setEpochs()

template<class InputType , class CacheType = float>

void shark::KernelSGDTrainer< InputType, CacheType >::setEpochs ( std::size_t value )

inline

Set the number of training epochs. A value of 0 indicates that the default of max(10, C) should be used.

Definition at line 204 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_epochs.

◆ setKernel()

template<class InputType , class CacheType = float>

void shark::KernelSGDTrainer< InputType, CacheType >::setKernel ( KernelType * kernel )

inline

set the kernel function

Definition at line 214 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::kernel(), and shark::KernelSGDTrainer< InputType, CacheType >::m_kernel.

◆ setParameterVector()

template<class InputType , class CacheType = float>

void shark::KernelSGDTrainer< InputType, CacheType >::setParameterVector ( RealVector const & newParameters )

inlinevirtual

Sets the vector of hyper-parameters.

Reimplemented from shark::IParameterizable< VectorType >.

Definition at line 245 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_C, shark::KernelSGDTrainer< InputType, CacheType >::m_kernel, shark::KernelSGDTrainer< InputType, CacheType >::m_unconstrained, shark::IParameterizable< VectorType >::numberOfParameters(), shark::IParameterizable< VectorType >::setParameterVector(), and SHARK_ASSERT.

◆ train()

template<class InputType , class CacheType = float>

void shark::KernelSGDTrainer< InputType, CacheType >::train	(	ClassifierType &	classifier,
		const LabeledData< InputType, unsigned int > &	dataset
	)

inline

Definition at line 140 of file KernelSGDTrainer.h.

References shark::KernelExpansion< InputType >::alpha(), shark::createBatch(), shark::Classifier< Model >::decisionFunction(), shark::random::discrete(), shark::Data< Type >::elements(), shark::AbstractLoss< LabelT, OutputT >::evalDerivative(), shark::random::globalRng, shark::LabeledData< InputT, LabelT >::inputs(), shark::LabeledData< InputT, LabelT >::labels(), shark::KernelSGDTrainer< InputType, CacheType >::m_C, shark::KernelSGDTrainer< InputType, CacheType >::m_cacheSize, shark::KernelSGDTrainer< InputType, CacheType >::m_epochs, shark::KernelSGDTrainer< InputType, CacheType >::m_kernel, shark::KernelSGDTrainer< InputType, CacheType >::m_loss, shark::KernelSGDTrainer< InputType, CacheType >::m_offset, shark::numberOfClasses(), shark::LabeledData< InputT, LabelT >::numberOfElements(), shark::KernelExpansion< InputType >::offset(), shark::PartlyPrecomputedMatrix< Matrix >::row(), and shark::KernelExpansion< InputType >::setStructure().

Referenced by main().

◆ trainOffset()

template<class InputType , class CacheType = float>

bool shark::KernelSGDTrainer< InputType, CacheType >::trainOffset ( ) const

inline

check whether the model to be trained should include an offset term

Definition at line 235 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_offset.

Member Data Documentation

◆ m_C

template<class InputType , class CacheType = float>

double shark::KernelSGDTrainer< InputType, CacheType >::m_C

protected

regularization parameter

Definition at line 261 of file KernelSGDTrainer.h.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::C(), shark::KernelSGDTrainer< InputType, CacheType >::parameterVector(), shark::KernelSGDTrainer< InputType, CacheType >::setC(), shark::KernelSGDTrainer< InputType, CacheType >::setParameterVector(), and shark::KernelSGDTrainer< InputType, CacheType >::train().

◆ m_cacheSize

template<class InputType , class CacheType = float>

std::size_t shark::KernelSGDTrainer< InputType, CacheType >::m_cacheSize

protected

Definition at line 267 of file KernelSGDTrainer.h.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::cacheSize(), shark::KernelSGDTrainer< InputType, CacheType >::setCacheSize(), and shark::KernelSGDTrainer< InputType, CacheType >::train().

◆ m_epochs

template<class InputType , class CacheType = float>

std::size_t shark::KernelSGDTrainer< InputType, CacheType >::m_epochs

protected

number of training epochs (sweeps over the data), or 0 for default = max(10, C)

Definition at line 264 of file KernelSGDTrainer.h.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::epochs(), shark::KernelSGDTrainer< InputType, CacheType >::setEpochs(), and shark::KernelSGDTrainer< InputType, CacheType >::train().

◆ m_kernel

template<class InputType , class CacheType = float>

KernelType* shark::KernelSGDTrainer< InputType, CacheType >::m_kernel

protected

pointer to kernel function

Definition at line 259 of file KernelSGDTrainer.h.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::kernel(), shark::KernelSGDTrainer< InputType, CacheType >::kernel(), shark::KernelSGDTrainer< InputType, CacheType >::numberOfParameters(), shark::KernelSGDTrainer< InputType, CacheType >::parameterVector(), shark::KernelSGDTrainer< InputType, CacheType >::setKernel(), shark::KernelSGDTrainer< InputType, CacheType >::setParameterVector(), and shark::KernelSGDTrainer< InputType, CacheType >::train().

◆ m_loss

template<class InputType , class CacheType = float>

const LossType* shark::KernelSGDTrainer< InputType, CacheType >::m_loss

protected

pointer to loss function

Definition at line 260 of file KernelSGDTrainer.h.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::train().

◆ m_offset

template<class InputType , class CacheType = float>

bool shark::KernelSGDTrainer< InputType, CacheType >::m_offset

protected

should the resulting model have an offset term?

Definition at line 262 of file KernelSGDTrainer.h.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::train(), and shark::KernelSGDTrainer< InputType, CacheType >::trainOffset().

◆ m_unconstrained

template<class InputType , class CacheType = float>

bool shark::KernelSGDTrainer< InputType, CacheType >::m_unconstrained

protected

should C be stored as log(C) as a parameter?

Definition at line 263 of file KernelSGDTrainer.h.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::isUnconstrained(), shark::KernelSGDTrainer< InputType, CacheType >::parameterVector(), and shark::KernelSGDTrainer< InputType, CacheType >::setParameterVector().

The documentation for this class was generated from the following file:

include/shark/Algorithms/Trainers/KernelSGDTrainer.h

Public Types

Public Member Functions

Protected Attributes

Detailed Description

Member Typedef Documentation

◆ ClassifierType

◆ ConstBatchInputReference

◆ KernelMatrixType

◆ KernelType

◆ LossType

◆ ModelType

◆ PartlyPrecomputedMatrixType

◆ QpFloatType

Constructor & Destructor Documentation

◆ KernelSGDTrainer()

Member Function Documentation

◆ C()

◆ cacheSize()

◆ epochs()

◆ isUnconstrained()

◆ kernel() [1/2]

◆ kernel() [2/2]

◆ name()

◆ numberOfParameters()

◆ parameterVector()

◆ setC()

◆ setCacheSize()

◆ setEpochs()

◆ setKernel()

◆ setParameterVector()

◆ train()

◆ trainOffset()

Member Data Documentation

◆ m_C

◆ m_cacheSize

◆ m_epochs

◆ m_kernel

◆ m_loss

◆ m_offset

◆ m_unconstrained