Adaptive Moment Estimation Algorithm (ADAM) More...

#include <shark/Algorithms/GradientDescent/Adam.h>

Inheritance diagram for shark::Adam< SearchPointType >:

Public Types
typedef AbstractObjectiveFunction< SearchPointType, double >	ObjectiveFunctionType

Public Types inherited from shark::AbstractSingleObjectiveOptimizer< PointType >
typedef base_type::SearchPointType	SearchPointType

typedef base_type::SolutionType	SolutionType

typedef base_type::ResultType	ResultType

typedef base_type::ObjectiveFunctionType	ObjectiveFunctionType

Public Types inherited from shark::AbstractOptimizer< PointType, double, SingleObjectiveResultSet< PointType > >
enum	Feature
	Models features that the optimizer requires from the objective function. More...

typedef PointType	SearchPointType

typedef double	ResultType

typedef SingleObjectiveResultSet< PointType >	SolutionType

typedef AbstractObjectiveFunction< PointType, ResultType >	ObjectiveFunctionType

typedef TypedFlags< Feature >	Features

typedef TypedFeatureNotAvailableException< Feature >	FeatureNotAvailableException

Public Member Functions
	Adam ()

std::string	name () const
	From INameable: return the class name.

void	init (ObjectiveFunctionType const &objectiveFunction, SearchPointType const &startingPoint)

double	eta () const
	get learning rate eta

void	setEta (double eta)
	set learning rate eta

double	beta1 () const
	get gradient averaging parameter beta1

void	setBeta1 (double beta1)
	set gradient averaging parameter beta1

double	beta2 () const
	get gradient averaging parameter beta2

void	setBeta2 (double beta2)
	set gradient averaging parameter beta2

double	epsilon () const
	get minimum noise estimate epsilon

void	setEpsilon (double epsilon)
	set minimum noise estimate epsilon

void	step (ObjectiveFunctionType const &objectiveFunction)
	Performs a step of the optimization.

virtual void	read (InArchive &archive)
	Read the component from the supplied archive.

virtual void	write (OutArchive &archive) const
	Write the component to the supplied archive.

Public Member Functions inherited from shark::AbstractSingleObjectiveOptimizer< PointType >
std::size_t	numInitPoints () const
	By default most single objective optimizers only require a single point.

virtual void	init (ObjectiveFunctionType const &function, std::vector< SearchPointType > const &initPoints)
	Initialize the optimizer for the supplied objective function using a set of initialisation points.

virtual void	init (ObjectiveFunctionType const &function, SearchPointType const &startingPoint)=0
	initializes the optimizer using a predefined starting point

virtual const SolutionType &	solution () const
	returns the current solution of the optimizer

Public Member Functions inherited from shark::AbstractOptimizer< PointType, double, SingleObjectiveResultSet< PointType > >
const Features &	features () const

virtual void	updateFeatures ()

bool	requiresValue () const

bool	requiresFirstDerivative () const

bool	requiresSecondDerivative () const

bool	canSolveConstrained () const

bool	requiresClosestFeasible () const

virtual	~AbstractOptimizer ()

virtual void	init (ObjectiveFunctionType const &function)
	Initialize the optimizer for the supplied objective function.

virtual void	step (ObjectiveFunctionType const &function)=0
	Carry out one step of the optimizer for the supplied objective function.

Public Member Functions inherited from shark::INameable
virtual	~INameable ()

Public Member Functions inherited from shark::ISerializable
virtual	~ISerializable ()
	Virtual d'tor.

void	load (InArchive &archive, unsigned int version)
	Versioned loading of components, calls read(...).

void	save (OutArchive &archive, unsigned int version) const
	Versioned storing of components, calls write(...).

	BOOST_SERIALIZATION_SPLIT_MEMBER ()

Additional Inherited Members
Protected Member Functions inherited from shark::AbstractOptimizer< PointType, double, SingleObjectiveResultSet< PointType > >
void	checkFeatures (ObjectiveFunctionType const &objectiveFunction)
	Convenience function that checks whether the features of the supplied objective function match with the required features of the optimizer.

Protected Attributes inherited from shark::AbstractSingleObjectiveOptimizer< PointType >
SolutionType	m_best
	Current solution of the optimizer.

Protected Attributes inherited from shark::AbstractOptimizer< PointType, double, SingleObjectiveResultSet< PointType > >
Features	m_features

Detailed Description

template<class SearchPointType = RealVector>
class shark::Adam< SearchPointType >

Adaptive Moment Estimation Algorithm (ADAM)

Performs SGD by using a long term average of the gradient as well as its second moment to adapt a step size for each coordinate.

Definition at line 47 of file Adam.h.

Member Typedef Documentation

◆ ObjectiveFunctionType

template<class SearchPointType = RealVector>

typedef AbstractObjectiveFunction<SearchPointType,double> shark::Adam< SearchPointType >::ObjectiveFunctionType

Definition at line 50 of file Adam.h.

Constructor & Destructor Documentation

◆ Adam()

template<class SearchPointType = RealVector>

shark::Adam< SearchPointType >::Adam ( )

inline

Definition at line 51 of file Adam.h.

References shark::AbstractOptimizer< PointType, double, SingleObjectiveResultSet< PointType > >::m_features, and shark::AbstractOptimizer< PointType, double, SingleObjectiveResultSet< PointType > >::REQUIRES_FIRST_DERIVATIVE.

Member Function Documentation

◆ beta1()

template<class SearchPointType = RealVector>

double shark::Adam< SearchPointType >::beta1 ( ) const

inline

get gradient averaging parameter beta1

Definition at line 91 of file Adam.h.

Referenced by shark::Adam< SearchPointType >::setBeta1().

◆ beta2()

template<class SearchPointType = RealVector>

double shark::Adam< SearchPointType >::beta2 ( ) const

inline

get gradient averaging parameter beta2

Definition at line 102 of file Adam.h.

Referenced by shark::Adam< SearchPointType >::setBeta2().

◆ epsilon()

template<class SearchPointType = RealVector>

double shark::Adam< SearchPointType >::epsilon ( ) const

inline

get minimum noise estimate epsilon

Definition at line 113 of file Adam.h.

Referenced by shark::Adam< SearchPointType >::setEpsilon().

◆ eta()

template<class SearchPointType = RealVector>

double shark::Adam< SearchPointType >::eta ( ) const

inline

get learning rate eta

Definition at line 80 of file Adam.h.

Referenced by shark::Adam< SearchPointType >::setEta().

◆ init()

template<class SearchPointType = RealVector>

void shark::Adam< SearchPointType >::init	(	ObjectiveFunctionType const &	objectiveFunction,
		SearchPointType const &	startingPoint
	)

inline

Definition at line 64 of file Adam.h.

References shark::AbstractOptimizer< PointType, double, SingleObjectiveResultSet< PointType > >::checkFeatures(), shark::AbstractObjectiveFunction< PointType, ResultT >::evalDerivative(), shark::AbstractSingleObjectiveOptimizer< PointType >::m_best, shark::AbstractObjectiveFunction< PointType, ResultT >::numberOfVariables(), shark::ResultSet< SearchPointT, ResultT >::point, SHARK_RUNTIME_CHECK, and shark::ResultSet< SearchPointT, ResultT >::value.

Referenced by main().

◆ name()

template<class SearchPointType = RealVector>

std::string shark::Adam< SearchPointType >::name ( ) const

inlinevirtual

From INameable: return the class name.

Reimplemented from shark::INameable.

Definition at line 61 of file Adam.h.

◆ read()

template<class SearchPointType = RealVector>

virtual void shark::Adam< SearchPointType >::read ( InArchive & archive )

inlinevirtual

Read the component from the supplied archive.

Parameters

[in,out] archive The archive to read from.

Reimplemented from shark::ISerializable.

Definition at line 143 of file Adam.h.

References shark::AbstractSingleObjectiveOptimizer< PointType >::m_best.

◆ setBeta1()

template<class SearchPointType = RealVector>

void shark::Adam< SearchPointType >::setBeta1 ( double beta1 )

inline

set gradient averaging parameter beta1

Definition at line 96 of file Adam.h.

References shark::Adam< SearchPointType >::beta1(), and SHARK_RUNTIME_CHECK.

◆ setBeta2()

template<class SearchPointType = RealVector>

void shark::Adam< SearchPointType >::setBeta2 ( double beta2 )

inline

set gradient averaging parameter beta2

Definition at line 107 of file Adam.h.

References shark::Adam< SearchPointType >::beta2(), and SHARK_RUNTIME_CHECK.

◆ setEpsilon()

template<class SearchPointType = RealVector>

void shark::Adam< SearchPointType >::setEpsilon ( double epsilon )

inline

set minimum noise estimate epsilon

Definition at line 118 of file Adam.h.

References shark::Adam< SearchPointType >::epsilon(), and SHARK_RUNTIME_CHECK.

◆ setEta()

template<class SearchPointType = RealVector>

void shark::Adam< SearchPointType >::setEta ( double eta )

inline

set learning rate eta

Definition at line 85 of file Adam.h.

References shark::Adam< SearchPointType >::eta(), and SHARK_RUNTIME_CHECK.

Referenced by main().

◆ step()

template<class SearchPointType = RealVector>

void shark::Adam< SearchPointType >::step ( ObjectiveFunctionType const & objectiveFunction )

inline

Performs a step of the optimization.

First the current guess for gradient and its second moment are updated using

\[ g_t = \beta_1 g_{t-1} + (1-\beta1) \frac{\partial}{\partial x} f(x_{t-1})\]

\[ v_t = \beta_2 v_{t-1} + (1-\beta2) (\frac{\partial}{\partial x} f(x_{t-1}))^2\]

The step is then performed as

\[ x_{t} = x_{t-1} - \eta * g_t *(sqrt(v_t) + \epsilon)^{-1} \]

where a slight step correction is used to remove the bias in the first few iterations where the means are close to 0.

Definition at line 131 of file Adam.h.

References shark::AbstractObjectiveFunction< PointType, ResultT >::evalDerivative(), shark::AbstractSingleObjectiveOptimizer< PointType >::m_best, shark::ResultSet< SearchPointT, ResultT >::point, shark::sqr(), and shark::ResultSet< SearchPointT, ResultT >::value.

Referenced by main().

◆ write()

template<class SearchPointType = RealVector>

virtual void shark::Adam< SearchPointType >::write ( OutArchive & archive ) const

inlinevirtual

Write the component to the supplied archive.

Parameters

[in,out] archive The archive to write to.

Reimplemented from shark::ISerializable.

Definition at line 156 of file Adam.h.

References shark::AbstractSingleObjectiveOptimizer< PointType >::m_best.

The documentation for this class was generated from the following file:

include/shark/Algorithms/GradientDescent/Adam.h

Public Types

Public Member Functions

Additional Inherited Members

Detailed Description

Member Typedef Documentation

◆ ObjectiveFunctionType

Constructor & Destructor Documentation

◆ Adam()

Member Function Documentation

◆ beta1()

◆ beta2()

◆ epsilon()

◆ eta()

◆ init()

◆ name()

◆ read()

◆ setBeta1()

◆ setBeta2()

◆ setEpsilon()

◆ setEta()

◆ step()

◆ write()